Learning Relation Alignment for Calibrated Cross-modal Retrieval

Abstract

Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions. The neglect of such relation consistency impairs the contextualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. In response, we present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions from the two modalities mutually via inter-modal alignment. The IAIS regularizer boosts the performance of prevailing models on Flickr30k and MS COCO datasets by a considerable margin, which demonstrates the superiority of our approach. 1

TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding

“…Video-Language Pre-trained Models. Benefitting from large-scale video-text datasets (Bain et al, 2021;Xue et al, 2021) and advances in Transformer model design (Gorti et al, 2022;Ren et al, 2021;Zellers et al, 2021;Wang et al, 2022a), pre-trained Video-Language Models (VidLMs) (Chen et al, 2022; have demonstrated impressive performance in video-language understanding tasks. VidLMs typically comprise a video encoder and a text encoder, which encode video-text pairs into a shared feature space to learn the semantic alignment between video and language.…”

“…( 2)-( 4)) on video-text pairs. Considering TESTA can aggregate tokens into objects, scenes, events, etc., training with fine-grained alignment functions (Ren et al, 2021;Wang et al, 2022c) could help some tasks like action localization and video object detection (Zhukov et al, 2019;Real et al, 2017), on which we will perform more explorations in future work.…”