Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrieval

Abstract

Image-sentence retrieval has attracted extensive research attention in multimedia and computer vision due to its promising application. The key issue lies in jointly learning the visual and textual representation to accurately estimate their similarity. To this end, the mainstream schema adopts an object-word based attention to calculate their relevance scores and refine their interactive representations with the attention features, which, however, neglects the context of the object representation on the inter-object relationship that matches the predicates in sentences. In this paper, we propose a Cross-modal Semantic Enhanced Interaction method, termed CMSEI for image-sentence retrieval, which correlates the intra-and inter-modal semantics between objects and words. In particular, we first design the intra-modal spatial and semantic graphs based reasoning to enhance the semantic representations of objects guided by the explicit relationships of the objects' spatial positions and their scene graph. Then the visual and textual semantic representations are refined jointly via the intermodal interactive attention and the cross-modal alignment. To correlate the context of objects with the textual context, we further refine the visual semantic representation via the cross-level object-sentence and word-image based interactive attention. Experimental results on seven standard evaluation metrics show that the proposed CMSEI outperforms the state-of-the-art and the alternative approaches on MS-COCO and Flickr30K benchmarks.

High-Accuracy Tomato Leaf Disease Image-Text Retrieval Method Utilizing LAFANet

“…A multi-head self-attention mechanism is then employed to effectively capture spatial context information within the image, resulting in features being extracted for these individual patches. To ensure semantic alignment between visual and textual data, we follow a similar strategy as in prior research [28,29]; the patch characteristics are aligned onto a unified semantic framework through fully connected layers. As a result, the identified attributes for each patch may be depicted as V = {v i /i = 1, ..., m, v i ∈ Rd}, where m denotes the quantity of patches per image.…”

Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining

“…Scene graphs and multimodality. The structural representations of scene graphs has been explored in the context of different V&L tasks, such as image-text retrieval (Johnson et al, 2015;Schuster et al, 2015;Schroeder and Tripathi, 2020;Ge et al, 2023), image captioning (Yao et al, 2018;Yang et al, 2019), and visual QA (Qian et al, 2022;Koner et al, 2021;Shi et al, 2019).…”