VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

Xu, Hongwu; Ghosh, Gargi; Huang, Po-Yao; Okhonko, Dmytro; Aghajanyan, Armen; Metze, Florian; Zettlemoyer, Luke; Feichtenhofer, Christoph

doi:10.18653/v1/2021.emnlp-main.544

Cited by 229 publications

(132 citation statements)

References 49 publications

Supporting

Mentioning

131

Contrasting

Order By: Relevance

“…Although they adopt GloVe [36] embeddings for query, the issues of feature gap are well alleviated. Considering recent advances in video-based vision-language pretraining (e.g., BVET [168], ActBERT [169], ClipBERT [170], and VideoCLIP [171]), dedicated or more effective feature extractors for TSGV are much expected.…”

Section: Effective Feature Extractor(s)mentioning

confidence: 99%

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

Zhang¹,

Sun²,

Wei³

et al. 2022

Preprint

View full text Add to dashboard Cite

Temporal sentence grounding in videos (TSGV), a.k.a., natural language video localization (NLVL) or video moment retrieval (VMR), aims to retrieve a temporal moment that semantically corresponds to a language query from an untrimmed video. Connecting computer vision and natural language, TSGV has drawn significant attention from researchers in both communities. This survey attempts to provide a summary of fundamental concepts in TSGV and current research status, as well as future research directions. As the background, we present a common structure of functional components in TSGV, in a tutorial style: from feature extraction from raw video and language query, to answer prediction of the target moment. Then we review the techniques for multimodal understanding and interaction, which is the key focus of TSGV for effective alignment between the two modalities. We construct a taxonomy of TSGV techniques and elaborate methods in different categories with their strengths and weaknesses. Lastly, we discuss issues with the current TSGV research and share our insights about promising research directions.

show abstract

Section: Effective Feature Extractor(s)mentioning

confidence: 99%

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

Zhang¹,

Sun²,

Wei³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Video-and-language Pre-training. Apart from the canonical pre-training tasks, such as masked language modeling (MLM) [10,26,30,35,48,57] and video-text matching (VTM) [30,35], several methods [35,37,52] apply contrastive learning on offline extracted visual features. Without adapting the visual backbone, their ability to align cross-modal features remain limited.…”

Section: Related Workmentioning

confidence: 99%

“…Existing sparse video-language pre-training models use either dot-product [3,37,39,52] or rely entirely on a transformer encoder [26,30,48,57] to model cross-modal interactions. However, since video and text features reside in different embedding spaces, such methods lead to less satisfactory alignment.…”

Section: Contrastive Video-text Alignmentmentioning

confidence: 99%

“…Note that we do not include comparison with the re- cent paper [36] which utilizes the powerful encoders from CLIP [44] (pretrained on 400M image-text pairs) and further post-trains them on HowTo100M [39]. Since [44] uses the same objective as VideoClip [52], the improvement over VideoClip validates the advantage of ALPRO pre-training.…”

Section: Evaluation On Video-text Retrievalmentioning

confidence: 99%

“…Despite their promising performance, current video-text pre-training models have several limitations. (1) The interaction between video and text features is commonly mod-eled trivially using either dot-product [3,37,39,52] or crossmodal transformer encoders [26,30,48,57]. However, features from individual modalities typically reside in different embedding spaces.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Align and Prompt: Video-and-Language Pre-training with Entity Prompts

Li¹,

Li²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Video-and-language pre-training has shown promising improvements on various downstream tasks. Most previous methods capture cross-modal interactions with a standard transformer-based multimodal encoder, not fully addressing the misalignment between unimodal video and text features. Besides, learning fine-grained visual-language alignment usually requires off-the-shelf object detectors to provide object information, which is bottlenecked by the detector's limited vocabulary and expensive computation cost.In this paper, we propose Align and Prompt: a new video-and-language pre-training framework (ALPRO), which operates on sparsely-sampled video frames and achieves more effective cross-modal alignment without explicit object detectors. First, we introduce a video-text contrastive (VTC) loss to align unimodal video-text features at the instance level, which eases the modeling of cross-modal interactions. Then, we propose a novel visually-grounded pre-training task, prompting entity modeling (PEM), which learns fine-grained alignment between visual region and text entity via an entity prompter module in a self-supervised way. Finally, we pretrain the video-and-language transformer models on large webly-source video-text pairs using the proposed VTC and PEM losses as well as two standard losses of masked language modeling (MLM) and video-text matching (VTM). The resulting pre-trained model achieves state-of-the-art performance on both text-video retrieval and videoQA, outperforming prior work by a substantial margin. Implementation and pre-trained models are available at https://github.com/salesforce/ALPRO.

show abstract

A review on multimodal zero‐shot learning

Cao

Sun

et al. 2023

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Multimodal learning provides a path to fully utilize all types of information related to the modeling target to provide the model with a global vision. Zero-shot learning (ZSL) is a general solution for incorporating prior knowledge into data-driven models and achieving accurate class identification. The combination of the two, known as multimodal ZSL (MZSL), can fully exploit the advantages of both technologies and is expected to produce models with greater generalization ability. However, the MZSL algorithms and applicationshave not yet been thoroughly investigated and summarized. This study fills this gap by providing an objective overview of MZSL's definition, typical algorithms, representative applications, and critical issues. This article will not only provide researchers in this field with a comprehensive perspective, but it will also highlight several promising research directions.

show abstract

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

Cited by 229 publications

References 49 publications

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

Align and Prompt: Video-and-Language Pre-training with Entity Prompts

A review on multimodal zero‐shot learning

Contact Info

Product

Resources

About