Knowledge Aware Semantic Concept Expansion for Image-Text Matching

Abstract

Image-text matching is a vital cross-modality task in artificial intelligence and has attracted increasing attention in recent years. Existing works have shown that learning semantic concepts is useful to enhance image representation and can significantly improve the performance of both image-to-text and text-to-image retrieval. However, existing models simply detect semantic concepts from a given image, which are less likely to deal with long-tail and occlusion concepts. Frequently co-occurred concepts in the same scene, e.g. bedroom and bed, can provide common-sense knowledge to discover other semantic-related concepts. In this paper, we develop a Scene Concept Graph (SCG) by aggregating image scene graphs and extracting frequently co-occurred concept pairs as scene common-sense knowledge. Moreover, we propose a novel model to incorporate this knowledge to improve image-text matching. Specifically, semantic concepts are detected from images and then expanded by the SCG. After learning to select relevant contextual concepts, we fuse their representations with the image embedding feature to feed into the matching module. Extensive experiments are conducted on Flickr30K and MSCOCO datasets, and prove that our model achieves state-of-the-art results due to the effectiveness of incorporating the external SCG.

A Deep Local and Global Scene-Graph Matching for Image-Text Retrieval

“…Many algorithms were invented to generate scene graphs from images [18,19,20,21]since this structure and the dataset about it was public. By virtue of them, some studies about image-text retrieval [15,16] have followed the scene graph approach recently. Shi et al [16] created a scene concept graph based on a popular scene graph dataset [22] and used it to expand the detected concepts in images to extract visual features.…”

“…In the image-text retrieval field, a node and an edge represent objects and the associations among objects detected in images or captions, as depicted in Figure 1 where the green rectangles are nodes in the graph indicating detected objects and the orange rectangles are edges illustrating the relationships between them. Recent research [15,16] could obtain better results in this retrieval area by utilizing these relations information. Hence there is a promise when applying this useful data structure in the retrieval field.…”

Context-Aware Multi-View Summarization Network for Image-Text Matching

“…Context information plays a pivotal role in understanding a sentence for many natural language processing tasks, such as neural machine translation [4,32], text summarization [29], and question answering [27]. Analogously, visual contextual relationships can contribute to obtaining fine-grained image region representations, which would benefit various tasks including image captioning [41], VQA [7], and image-text matching [17,31,35,40]. To exploit visual and textual context and capture implicit relations among intra-modal fragments, researchers have presented some structured models for different multi-modal tasks.…”

“…In the field of image-text matching, Li et al [17] performed local-global semantic reasoning by using Graph Convolutional Network (GCN) and Gated Recurrent Unit. For learning comprehensive representations, Wang et al [35] and Shi et al [31] refined visual relationships by leveraging external scene graphs [13]. Wu et al [40] considered fragment relations in images and texts to obtain self-attention embeddings, acquiring promising intra-modal context modeling.…”

“…To justify the effectiveness of our proposal, we compared our proposed CAMERA with the following state-of-the-art baselines: VQA [20], sm-LSTM [11], VSE++ [5], CMPM + CMPC [44], SCAN [16], SCG [31], TIMAM [30], SAEM [40], CAMP [37], BFAN [21], PFAN [36], SAN [12], VSRN [17], and SGM [35]. Note that we directly quoted the results from their original papers.…”