Consensus-Aware Visual-Semantic Embedding for Image-Text Matching

Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval

“…In contrast to previous studies, SMFEA models the relation structure of intra-modal fragments/words by the use of a fixed contextual structure and aligns two modalities into a joint embedding space in terms of semantics and structure. The most relevant existing work to ours is [30], which aligns the visual and textual representations through measuring the consistency of the corresponding concepts in each modality. However unlike [30], SMFEA approaches this in a novel way by exploiting the learned multi-modal semantic trees to…”

“…The most relevant existing work to ours is [30], which aligns the visual and textual representations through measuring the consistency of the corresponding concepts in each modality. However unlike [30], SMFEA approaches this in a novel way by exploiting the learned multi-modal semantic trees to…”

“…By aligning the inter-modal semantics and structure consistently, the joint embedding space is obtained to reduce the heterogeneous (inter-modality) semantic gap. Doing so allows us to provide more robustness than [30], which also improves the interpretability of the model.…”

Bidirectional Focused Semantic Alignment Attention Network for Cross-Modal Retrieval

“…Specifically, He et al [3] created a new fine-grained cross-media dataset, which contains four modal information (image, text, video, and audio). The collection of cross-modal datasets is more difficult, researchers usually analyze retrieval tasks in two modalities [4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20]. Zhang et al [4] proposed an aggregating the global context to mine the semantic information of image modality and text modality.…”

“…Wei et al [8] proposed the universal weighting metric learning method. In addition, some researchers have explored cross-modal retrieval algorithms from attention mechanism, graph reasoning, and loss function optimization [9][10][11][12][13][14][15][16][17][18][19][20]. These studies promote the development of cross-modal retrieval tasks.…”