HunYuan_tvr for Text-Video Retrieval

Abstract

Text-Video Retrieval plays an important role in multi-modal understanding and has attracted increasing attention in recent years. Most existing methods focus on constructing contrastive pairs between whole videos and complete caption sentences, while ignoring fine-grained cross-modal relationships, e.g., short clips and phrases or single frame and word. In this paper, we propose a novel method, named HunYuan_tvr, to explore hierarchical cross-modal interactions by simultaneously exploring video-sentence, clip-phrase, and frame-word relationships. Considering intrinsic semantic relations between frames, HunYuan_tvr first performs self-attention to explore frame-wise correlations and adaptively clusters correlated frames into clip-level representations. Then, the clip-wise correlation is explored to aggregate clip representations into a compact one to describe the video globally. In this way, we can construct hierarchical video representations for frame-clip-video granularities, and also explore word-wise correlations to form word-phrase-sentence embeddings for the text modality. Finally, hierarchical contrastive learning is designed to explore cross-modal relationships, i.e., frame-word, clip-phrase, and video-sentence, which enables HunYuan_tvr to achieve a comprehensive multi-modal understanding. Further boosted by adaptive label denoising and marginal sample enhancement, HunYuan_tvr obtains new state-of-the-art results on various benchmarks, e.g.,

Token Mixing: Parameter-Efficient Transfer Learning from Image-Language to Video-Language

“…Most existing approaches (Cao et al 2022;Chen, Liu, and Albanie 2021;Wang et al 2022b;Liu et al 2021;Yang et al 2021;Suin and Rajagopalan 2020;Hou et al 2020;Chen, Liu, and Albanie 2021;Ryu et al 2021;Lin, Gan, and Wang 2021;Xu et al 2019;Chen et al 2019;Zhang, Song, and Jin 2022) explore task specific modules for different tasks. For example, for the video retrieval task, HiT (Liu et al 2021) and Hunyuan tvr (Min et al 2022) use a hierarchical matching strategy for crossmodal interaction. For the video captioning task, Open-book Apart from task specific modules, we believe a powerful video encoder can bring performance gain in any videolanguage tasks.…”

“…For video captioning, we compare with ORG-TRL (Zhang et al 2020) method vs. 64 in SwinBERT (Lin et al 2022)). For video retrieval, we compare with TS2-Net (Liu et al 2022), Hunyuan tvr (Min et al 2022), and video-language pre-trained model OmniVL (Wang et al 2022a). All methods use the same frame resolution.We observe that with minor adaptions and only a small part of parameters updated, our method still achieves comparable performance.…”