How a General-Purpose Commonsense Ontology can Improve Performance of Learning-Based Image Retrieval

Abstract

The knowledge representation community has built general-purpose ontologies which contain large amounts of commonsense knowledge over relevant aspects of the world, including useful visual information, e.g.: "a ball is used by a football player", "a tennis player is located at a tennis court". Current state-of-the-art approaches for visual recognition do not exploit these rule-based knowledge sources. Instead, they learn recognition models directly from training examples. In this paper, we study how general-purpose ontologiesspecifically, MIT's ConceptNet ontology-can improve the performance of state-of-the-art vision systems. As a testbed, we tackle the problem of sentence-based image retrieval. Our retrieval approach incorporates knowledge from ConceptNet on top of a large pool of object detectors derived from a deep learning technique. In our experiments, we show that ConceptNet can improve performance on a common benchmark dataset. Key to our performance is the use of the ESPGAME dataset to select visually relevant relations from ConceptNet. Consequently, a main conclusion of this work is that general-purpose commonsense ontologies improve performance on visual reasoning tasks when properly filtered to select meaningful visual relations.

Object-Action Association Extraction from Knowledge Graphs

“…Some recent studies that apply this method, as is or with small variations, are [16,4,5,15]. In fact, they also focus on inferring object-action relations ( [16,5]) and on object identification ( [4,15]).…”

“…Extracting commonsense knowledge from problem-agnostic repositories has been applied to a diversity of AI-related domains to solve various problems. The authors of [4] rely upon ConceptNet to identify word similarities, which they then use in order to improve the performance of sentence-based image retrieval algorithms. A more elaborate use of KGs is presented in [5], where the authors approach the problem of zero-shot label learning in images by creating KGs based on labels detected visually and on correlations found in external sources.…”

“…Before presenting the methods we evaluated to solve the aforementioned problem, we first describe how we can create the graph G from a given set of labels L that refer to real-world objects or actions. We extract the object and action labels from the Something-Something Dataset 4 , a dataset that is commonly used by the Machine Vision community (see Section 4.1 for more details). Note, however, that any set of object and action labels can be used to create G. For every label l i ∈ L, we generate a graph S i , by appending information relevant to l i from ConceptNet [1] and WordNet [2] in two steps.…”

A Knowledge Retrieval Framework for Household Objects and Actions with External Knowledge

“…Our semantic matching algorithm was mostly inspired by the works of Young et al [35], and Icarte et al [12] where they use CS knowledge from the web ontologies DBpedia, ConceptNet, and WordNet to find the label of unknown objects. As well as from the studies [6,36], where the label of the room can be understood through the objects that the cognitive robotic system perceived from its vision module.…”

“…The libraries Request and NLTK 3 offer web APIs for all three aforementioned ontologies. Similar methods can be found in [12,35], where they also exploit the CS knowledge existing in web ontologies. Algorithm 1 starts by getting as input any word that is part of the English language; we check this by obtaining the WordNet entity, line 3.…”