Dig into Multi-modal Cues for Video Retrieval with Hierarchical Alignment

Wang, Wenzhe; Zhang, Mengdan; Chen, Runnan; Cai, Guangming; Zhou, Penghao; Peng, Pai; Guo, Xiaowei; Wu, Jian; Sun, Xing

doi:10.24963/ijcai.2021/154

Cited by 17 publications

(8 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Text-Video Retrieval is to find the most semantic-relevant video given a text query (text → video). Early research devotes to distilling knowledge from "expert" models based on offline-extracted single-modality features [8,13,26,44]. However, the performance is far from satisfactory due to the significant domain gap.…”

Section: Related Workmentioning

confidence: 99%

UATVR: Uncertainty-Adaptive Text-Video Retrieval

Fang¹,

wu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

With the explosive growth of web videos and emerging large-scale vision-language pre-training models, e.g., CLIP, retrieving videos of interest with text instructions has attracted increasing attention. A common practice is to transfer text-video pairs to the same embedding space and craft cross-modal interactions with certain entities in specific granularities for semantic correspondence. Unfortunately, the intrinsic uncertainties of optimal entity combinations in appropriate granularities for cross-modal queries are understudied, which is especially critical for modalities with hierarchical semantics, e.g., video, text, etc. In this paper, we propose an Uncertainty-Adaptive Text-Video Retrieval approach, termed UATVR, which models each lookup as a distribution matching procedure. Concretely, we add additional learnable tokens in the encoders to adaptively aggregate multi-grained semantics for flexible highlevel reasoning. In the refined embedding space, we represent text-video pairs as probabilistic distributions where prototypes are sampled for matching evaluation. Comprehensive experiments on four benchmarks justify the superiority of our UATVR, which achieves new state-of-theart results on MSR-VTT (50.8%), VATEX (64.5%), MSVD (49.7%), and DiDeMo (45.8%). The code is available in supplementary materials and will be released publicly soon.

show abstract

Section: Related Workmentioning

confidence: 99%

UATVR: Uncertainty-Adaptive Text-Video Retrieval

Fang¹,

wu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…With the release of the large-scale instructional dataset HowTo100M, VTP has spurred significant interest in the community. Overall, the mainstream methods can be broadly classified into two classes: 1) Generative methods: Several methods [28,34,50,11,20,56,31,55] try to extend BERT [53] to the cross-modal domain, i.e., they accept both visual and textual tokens as input and perform the masked-token prediction task. 2) Discriminative methods.…”

Section: Related Workmentioning

confidence: 99%

LocVTP: Video-Text Pre-training for Temporal Localization

Cao¹,

Yang²,

Weng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video-Text Pre-training (VTP) aims to learn transferable representations for various downstream tasks from large-scale web videos. To date, almost all existing VTP methods are limited to retrieval-based downstream tasks, e.g., video retrieval, whereas their transfer potentials on localization-based tasks, e.g., temporal grounding, are underexplored. In this paper, we experimentally analyze and demonstrate the incompatibility of current VTP methods with localization tasks, and propose a novel Localization-oriented Video-Text Pre-training framework, dubbed as LocVTP. Specifically, we perform the fine-grained contrastive alignment as a complement to the coarse-grained one by a clipword correspondence discovery scheme. To further enhance the temporal reasoning ability of the learned feature, we propose a context projection head and a temporal aware contrastive loss to perceive the contextual relationships. Extensive experiments on four downstream tasks across six datasets demonstrate that our LocVTP achieves state-of-the-art performance on both retrieval-based and localization-based tasks. Furthermore, we conduct comprehensive ablation studies and thorough analyses to explore the optimum model designs and training strategies. Codes are available at https://github.com/mengcaopku/LocVTP.

show abstract

“…Video-Language Pre-training. Early VLP methods [30,56,45,53,17,48] introduce pretrained models on other tasks to pre-extract video representations. Some of them [30,56,45] utilize action recognition backbones [15,19] to pre-extract video representations.…”

Section: Related Workmentioning

confidence: 99%

“…These backbones are designed with 2D [20] and 3D [19] CNNs to capture spatial and temporal information in videos. Others [53,35,17,48] fuse multiple "Experts" that are trained on different modalities, such as audio classification [21], OCR [17], image classification [22] and so on, to fully exploit cross-modal high-level semantics in videos. Recently, end-to-end models [37,29,4,54,16] are proposed .…”

Section: Related Workmentioning

confidence: 99%

Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval

Cai¹,

Ge²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recent dominant methods for video-language pre-training (VLP) learn transferable representations from the raw pixels in an end-to-end manner to achieve advanced performance on downstream video-language tasks. Despite the impressive results, VLP research becomes extremely expensive with the need for massive data and a long training time, preventing further explorations. In this work, we revitalize region features of sparsely sampled video clips to significantly reduce both spatial and temporal visual redundancy towards democratizing VLP research at the same time achieving state-of-the-art results. Specifically, to fully explore the potential of region features, we introduce a novel bidirectional region-word alignment regularization that properly optimizes the fine-grained relations between regions and certain words in sentences, eliminating the domain/modality disconnections between pre-extracted region features and text. Extensive results of downstream text-to-video retrieval and video question answering tasks on seven datasets demonstrate the superiority of our method on both effectiveness and efficiency, e.g., our method achieves competing results with 80% fewer data and 85% less pre-training time compared to the most efficient VLP method so far [29]. The code will be available at https://github.com/showlab/DemoVLP.

show abstract

Dig into Multi-modal Cues for Video Retrieval with Hierarchical Alignment

Cited by 17 publications

References 17 publications

UATVR: Uncertainty-Adaptive Text-Video Retrieval

UATVR: Uncertainty-Adaptive Text-Video Retrieval

LocVTP: Video-Text Pre-training for Temporal Localization

Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval

Contact Info

Product

Resources

About