Align and Prompt: Video-and-Language Pre-training with Entity Prompts

Abstract

Video-and-language pre-training has shown promising improvements on various downstream tasks. Most previous methods capture cross-modal interactions with a standard transformer-based multimodal encoder, not fully addressing the misalignment between unimodal video and text features. Besides, learning fine-grained visual-language alignment usually requires off-the-shelf object detectors to provide object information, which is bottlenecked by the detector's limited vocabulary and expensive computation cost.In this paper, we propose Align and Prompt: a new video-and-language pre-training framework (ALPRO), which operates on sparsely-sampled video frames and achieves more effective cross-modal alignment without explicit object detectors. First, we introduce a video-text contrastive (VTC) loss to align unimodal video-text features at the instance level, which eases the modeling of cross-modal interactions. Then, we propose a novel visually-grounded pre-training task, prompting entity modeling (PEM), which learns fine-grained alignment between visual region and text entity via an entity prompter module in a self-supervised way. Finally, we pretrain the video-and-language transformer models on large webly-source video-text pairs using the proposed VTC and PEM losses as well as two standard losses of masked language modeling (MLM) and video-text matching (VTM). The resulting pre-trained model achieves state-of-the-art performance on both text-video retrieval and videoQA, outperforming prior work by a substantial margin. Implementation and pre-trained models are available at https://github.com/salesforce/ALPRO.

OMG: Observe Multiple Granularities for Natural Language-Based Vehicle Retrieval

“…Recently, with the successful migration of Transformer [43] from natural language processing to computer vision, (ViT [5], Swin-Transformer [24], etc. ), the mainstream methods [6,9,16,17,19,22,26,35,44,45] on video-language retrieval tasks begin to utilize Transformer as encoders for both of the video and the natural language. HERO [17] and ALPRO [16] explore to boost the video-text alignment via the large-scale pre-training tasks.…”

“…), the mainstream methods [6,9,16,17,19,22,26,35,44,45] on video-language retrieval tasks begin to utilize Transformer as encoders for both of the video and the natural language. HERO [17] and ALPRO [16] explore to boost the video-text alignment via the large-scale pre-training tasks. HiT [22] further proposed a hierarchical model with momentum contrast for videotext retrieval.…”

Clover: Towards A Unified Video-Language Alignment and Fusion Model

“…Video-language pre-training (VidL) aims to learn generalizable multi-modal models from largescale video-text samples so as to better solve various challenging video-language understanding tasks, such as text-video retrieval [28,4,1,43] and video question answering [12,36,40]. Recent studies [16,17,7,9,44,47] have shown that VidL leads to significant performance improvement and achieves state-of-the-art results on various downstream text-video retrieval and video question answering (VQA) benchmarks.…”

“…Building a unified model capable of solving various video-language tasks is a long-standing challenge for machine learning research. A few recent works [7,17] attempt to learn a unified VidL model for both tasks, which uses the multi-modal encoder to conduct text-video retrieval. However, the model requires an exhaustive pair-wise comparison between the query texts and gallery videos.…”

HunYuan_tvr for Text-Video Retrieval

“…However, a natural semantic gap between two modalities, i.e., video and text, raises a great challenge, which hinders industrial-level applications of TVR. To this end, recent methods target to distill cross-modal knowledge from largescale pretraining experts [19,12,17,28] and leverage cross-modal contrastive learning to explore both intra-modal representation and cross-modal interaction.…”