Boosting Video-Text Retrieval with Explicit High-Level Semantics

Abstract

Recently, masked video modeling has been widely explored and significantly improved the model's understanding ability of visual regions at a local level. However, existing methods usually adopt random masking and follow the same reconstruction paradigm to complete the masked regions, which do not leverage the correlations between cross-modal content. In this paper, we present MAsk for Semantics COmpleTion (MASCOT) based on semantic-based masked modeling. Specifically, after applying attention-based video masking to generate highinformed and low-informed masks, we propose Informed Semantics Completion to recover masked semantics information. The recovery mechanism is achieved by aligning the masked content with the unmasked visual regions and corresponding textual context, which makes the model capture more text-related details at a patch level. Additionally, we shift the emphasis of reconstruction from irrelevant backgrounds to discriminative parts to ignore regions with low-informed masks. Furthermore, we design dualmask co-learning to incorporate video cues under different masks and learn more aligned video representation. Our MASCOT performs state-of-the-art performance on four major text-video retrieval benchmarks, including MSR-VTT, LSMDC, ActivityNet, and DiDeMo. Extensive ablation studies demonstrate the effectiveness of the proposed schemes.

Video-Text Retrieval by Supervised Multi-Space Multi-Grained Alignment

“…To show the empirical efficiency of our SUMA, we train models on MSR-VTT (Xu et al, 2016), MSVD (Chen and Dolan, 2011), and Activi-tyNet (Fabian Caba Heilbron and Niebles, 2015). For a fair comparison, we only compare our methods with methods that are based on CLIP (Radford et al, 2021), i.e., Clip4Clip (Luo et al, 2022), CLIP2TV (Gao et al, 2021), X-CLIP , DiscreteCodebook (Liu et al, 2022a), TS2-Net (Liu et al, 2022b), CLIP2Video (Park et al, 2022), VCM , HiSE (Wang et al, 2022a), Align&Tell (Wang et al, 2022b), Center-CLIP (Zhao et al, 2022), and X-Pool (Gorti et al, 2022). Implementation details and evaluation protocols are deferred to the Appendix.…”

Video-Text Retrieval by Supervised Sparse Multi-Grained Learning

“…To show the empirical efficiency of our S3MA, we train it on MSR-VTT (Xu et al, 2016), MSVD Dolan, 2011), andActivi-tyNet (Fabian Caba Heilbron andNiebles, 2015). We compare with VLM (Xu et al, 2021a), HERO (Li et al, 2020a), VideoCLIP (Xu et al, 2021b), EvO (Shvetsova et al, 2022), OA-Trans (Wang et al, 2022a), RaP (Wu et al, 2022), LiteVL , NCL (Park et al, 2022b), TABLE (Chen et al, 2023), VOP (Huang et al, 2023), Clip4Clip (Luo et al, 2022), X-CLIP (Ma et al, 2022a), DiscreteCodebook (Liu et al, 2022a), TS2-Net (Liu et al, 2022b), VCM , HiSE (Wang et al, 2022b), Align&Tell (Wang et al, 2022c), Center-CLIP (Zhao et al, 2022), andX-Pool (Gorti et al, 2022). Implementation details and evaluation protocols are deferred to the Appendix.…”