Finding Moments in Video Collections Using Natural Language

Escorcia, Víctor; Soldan, Mattia; Šivic, Josef; Ghanem, Bernard; Russell, Bryan

doi:10.48550/arxiv.1907.12763

Cited by 22 publications

(63 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DiDeMo [4], ActivityNet Captions [32], and Charades-STA [17] are among the benchmarks for this task. Finally, VCMR came even more recently from [15] and [37] which extends the searching scope of SVMR to the entire video corpus.…”

Section: Appendix 61 Related Workmentioning

confidence: 99%

CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning

Zhou,

Liu,

Cheng

et al. 2021

Preprint

View full text Add to dashboard Cite

This work concerns video-language pre-training and representation learning. In this now ubiquitous training scheme, a model first performs pre-training on paired videos and text (e.g., video clips and accompanied subtitles) from a large uncurated source corpus, before transferring to specific downstream tasks. This two-stage training process inevitably raises questions about the generalization ability of the pre-trained model, which is particularly pronounced when a salient domain gap exists between source and target data (e.g., instructional cooking videos vs. movies). In this paper, we first bring to light the sensitivity of pre-training objectives (contrastive vs. reconstructive) to domain discrepancy. Then, we propose a simple yet effective framework, CUPID, to bridge this domain gap by filtering and adapting source data to the target data, followed by domain-focused pre-training. Comprehensive experiments demonstrate that pre-training on a considerably small subset of domain-focused data can effectively close the source-target domain gap and achieve significant performance gain, compared to random sampling or even exploiting the full pre-training dataset. CU-PID yields new state-of-the-art performance across multiple video-language and video tasks, including text-to-video retrieval [72,37], video question answering [36], and video captioning [72], with consistent performance lift over different pre-training methods.

show abstract

Section: Appendix 61 Related Workmentioning

confidence: 99%

CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning

Zhou,

Liu,

Cheng

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…4.2.1 Large-scale video corpus moment retrieval. Large-scale video corpus moment retrieval (VCMR) is a research direction extended from TSGV that has been explored over the past few years [15,32,77,79]. It has more application value since it can retrieve the target segment semantically corresponding to a given text query from a large-scale video corpus (i.e., a collection of untrimmed and unsegmented videos) rather than from a single video.…”

Section: Promising Research Directionsmentioning

confidence: 99%

“…Escorcia et al [15] first extend TSGV to VCMR, introducing a model named Clip Alignment with Language (CAL) to align the query feature with a sequence of uniformly partitioned clips for moment composing. Lei et al [32] introduce a new dataset for VCMR called TVR, which is comprised of videos and their associated subtitle texts.…”

Section: Promising Research Directionsmentioning

confidence: 99%

A Survey on Temporal Sentence Grounding in Videos

Lan¹,

Yuan²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Temporal sentence grounding in videos (TSGV), which aims to localize one target segment from an untrimmed video with respect to a given sentence query, has drawn increasing attentions in the research community over the past few years. Different from the task of temporal action localization, TSGV is more flexible since it can locate complicated activities via natural languages, without restrictions from predefined action categories. Meanwhile, TSGV is more challenging since it requires both textual and visual understanding for semantic alignment between two modalities (i.e., text and video). In this survey, we give a comprehensive overview for TSGV, which i) summarizes the taxonomy of existing methods, ii) provides a detailed description of the evaluation protocols (i.e., datasets and metrics) to be used in TSGV, and iii) in-depth discusses potential problems of current benchmarking designs and research directions for further investigations. To the best of our knowledge, this is the first systematic survey on temporal sentence grounding. More specifically, we first discuss existing TSGV approaches by grouping them into four categories, i.e., two-stage methods, end-to-end methods, reinforcement learning-based methods, and weakly supervised methods. Then we present the benchmark datasets and evaluation metrics to assess current research progress. Finally, we discuss some limitations in TSGV through pointing out potential problems improperly resolved in the current evaluation protocols, which may push forwards more cutting edge research in TSGV. Besides, we also share our insights on several promising directions, including three typical tasks with new and practical settings based on TSGV.

show abstract

“…Video Corpus Moment Retrieval (VCMR) [1] is a new video-text retrieval task which aims to retrieve the most relevant moments from a large video corpus instead of from a single video. The text-based VCMR can be decomposed into two sub-tasks: video retrieval (VR) and single video moment retrieval (SVMR).…”

Section: Introductionmentioning

confidence: 99%

Coarse to Fine: Video Retrieval before Moment Localization

Gao,

Liu,

Liu

2021

Preprint

View full text Add to dashboard Cite

The current state-of-the-art methods for video corpus moment retrieval (VCMR) often use similarity-based feature alignment approach for the sake of convenience and speed. However, late fusion methods like cosine similarity alignment are unable to make full use of the information from both query texts and videos. In this paper, we combine feature alignment with feature fusion to promote the performance on VCMR.

show abstract

Finding Moments in Video Collections Using Natural Language

Cited by 22 publications

References 37 publications

CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning

CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning

A Survey on Temporal Sentence Grounding in Videos

Coarse to Fine: Video Retrieval before Moment Localization

Contact Info

Product

Resources

About