Cross-Modal Hybrid Feature Fusion for Image-Sentence Matching

Abstract

Image-sentence matching is a challenging task in the field of language and vision, which aims at measuring the similarities between images and sentence descriptions. Most existing methods independently map the global features of images and sentences into a common space to calculate the image-sentence similarity. However, the image-sentence similarity obtained by these methods may be coarse as (1) an intermediate common space is introduced to implicitly match the heterogeneous features of images and sentences in a global level, and (2) only the inter-modality relations of images and sentences are captured while the intra-modality relations are ignored. To overcome the limitations, we propose a novel Cross-Modal Hybrid Feature Fusion (CMHF) framework for directly learning the image-sentence similarity by fusing multimodal features with inter- and intra-modality relations incorporated. It can robustly capture the high-level interactions between visual regions in images and words in sentences, where flexible attention mechanisms are utilized to generate effective attention flows within and across the modalities of images and sentences. A structured objective with ranking loss constraint is formed in CMHF to learn the image-sentence similarity based on the fused fine-grained features of different modalities bypassing the usage of intermediate common space. Extensive experiments and comprehensive analysis performed on two widely used datasets—Microsoft COCO and Flickr30K—show the effectiveness of the hybrid feature fusion framework in CMHF, in which the state-of-the-art matching performance is achieved by our proposed CMHF method.

Image–Text Cross-Modal Retrieval with Instance Contrastive Embedding

“…The image-text cross-modal retrieval task is designed to explore the correspondence between image and text. The existing matching methods can be roughly divided into two categories: graph-free paradigm [6,7,[10][11][12][13][14][18][19][20][21][22][23][24][25][26][27] and graph-based paradigm [8,9,[15][16][17][29][30][31][32][33].…”

“…While these methods show promising results in image-text cross-modal retrieval, they mainly embed global feature representations and overlook the fine-grained semantic associations between image and text. To tackle this limitation, recent research concentrates on learning correspondences between image regions and text words, achieving semantic coverage from coarse-to-fine [11][12][13][14]. For instance, Xu et al [11] propose a cross-modal hybrid feature fusion method to capture interactions between image and text, which learns image-text similarity by fusing feature representation of intra-and inter-modality, providing robust semantic interactions between image regions and text words.…”

“…To tackle this limitation, recent research concentrates on learning correspondences between image regions and text words, achieving semantic coverage from coarse-to-fine [11][12][13][14]. For instance, Xu et al [11] propose a cross-modal hybrid feature fusion method to capture interactions between image and text, which learns image-text similarity by fusing feature representation of intra-and inter-modality, providing robust semantic interactions between image regions and text words. Another work by Lan et al [13] proposes a multi-level matching network model that incorporates multi-level similarity between image and text via adaptive matching integration strategies.…”

“…However, these methods only capture the rough semantic correlation between different modalities and fail to describe the local semantic correspondence between image regions and text words effectively. To address this limitation, fine-grained cross-modal retrieval methods [3,8,9,[11][12][13][14][15][16][17] have been proposed for modelling the local similarity between image regions and text words. Currently, fine-grained image-text cross-modal retrieval methods can be roughly divided into two categories: (1) Graph-free paradigm [6,7,[10][11][12][13][14][18][19][20][21][22][23][24][25][26][27]: These methods typically encode multi-level feature representations using the output of the last layer of the encoder and then fuse these multi-level similarities to obtain the final cross-modal similarity.…”

“…To address this limitation, fine-grained cross-modal retrieval methods [3,8,9,[11][12][13][14][15][16][17] have been proposed for modelling the local similarity between image regions and text words. Currently, fine-grained image-text cross-modal retrieval methods can be roughly divided into two categories: (1) Graph-free paradigm [6,7,[10][11][12][13][14][18][19][20][21][22][23][24][25][26][27]: These methods typically encode multi-level feature representations using the output of the last layer of the encoder and then fuse these multi-level similarities to obtain the final cross-modal similarity. Additionally, region-level features from visual target detectors (e.g., Faster R-CNN [28]) are employed to establish semantic alignment between image regions and text words.…”