Robots with language: Multi-label visual recognition using NLP

There has been a recent interest in utilizing contextual knowledge to improve multi-label visual recognition for intelligent agents like robots. Natural Language Processing (NLP) can give us labels, the correlation of labels, and the ontological knowledge about them, so we can automate the acquisition of contextual knowledge. In this paper we show how to use tools from NLP in conjunction with Vision to improve visual recognition. There are two major approaches: First, different language databases organize words according to various semantic concepts. Using these, we can build special purpose databases that can predict the labels involved given a certain context. Here we build a knowledge base for the purpose of describing common daily activities. Second, statistical language tools can provide the correlations of different labels. We show a way to learn a language model from large corpus data that exploits these correlations and propose a general optimization scheme to integrate the language model into the system. Experiments conducted on three multi-label everyday recognition tasks support the effectiveness and efficiency of our approach, with significant gains in recognition accuracies when correlation information is used.

Authors

• Yezhou Yang
• Ching Lik Teo
• Cornelia Fermüller
• Yiannis Aloimonos

Keywords

• Correlation
• Visualization
• Robots
• Accuracy
• Knowledge based systems
• Databases
• Hidden Markov models
• Visual Recognition
• Knowledge Base
• Language Processing
• Recognition Task
• Recognition Accuracy
• Common Activities
• Language Model
• Intelligence Agencies
• Large Corpus
• Language Tools
• Computer Vision
• Visual Processing
• Hidden Markov Model
• Object Recognition
• Object Classification
• Graphical Model
• Action Recognition
• Object Properties
• Ground Truth Labels
• Reasoning Process
• Language Information
• Computational Linguistics
• Object Appearance
• Hand Trajectory
• Scene Classification
• Labeling Task
• SVM Classifier
• Natural Language Processing Tools
• Action Labels
• WordNet