Weakly Supervised Video Moment Retrieval From Text Queries

Mithun, Niluthpol Chowdhury; Shrivastava, Paul; Roy-Chowdhury, Amit K.

doi:10.1109/cvpr.2019.01186

Cited by 158 publications

(87 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TGA. (Mithun, Paul, and Roy-Chowdhury 2019) The TGA method proposes a weakly-supervised joint visual-semantic embedding framework for moment retrieval, and utilizes the latent alignment for localization during inference.…”

Section: Compared Methodsmentioning

confidence: 99%

“…Though those methods achieve good performance, they still suffer from collecting a large amount of manually labelled temporal annotations. Some works (Bojanowski et al 2015;Duan et al 2018;Mithun, Paul, and 2019) also study this task in a weakly-supervised setting. The method proposed in (Bojanowski et al 2015) consider the task of aligning a video with a set of temporal ordered sentences, in which temporal ordering can be seen as additional constraint and supervision.…”

Section: Related Workmentioning

confidence: 99%

“…The method proposed in (Duan et al 2018) decomposes the problem of weaklysupervised dense event captioning in videos (WS-DEC) into a cycle of dual problems: caption generation and moment retrieval and explores the one-to-one correspondence between the temporal segment and event caption, and has a complex training pipeline such as pre-training and alternating training. The Text-Guided Attention (TGA) (Mithun, Paul, and Roy-Chowdhury 2019) proposes to learn a joint visualsemantic representations and utilizes the attention score as the alignment between video frames and query.…”

Section: Related Workmentioning

confidence: 99%

“…This motivates us to develop a weakly-supervised method for moment retrieval that needs only coarse video-level annotations for training. Existing weakly-supervised method in (Mithun, Paul, and Roy-Chowdhury 2019) proposes to learn a joint visual-text embedding, and utilizes the latent alignment produced by intermediate Text-Guided Attention (TGA) to localize the target moment. However, the latent attention weights without extra supervision usually focus on the most discriminative but small regions (Singh and Lee 2017) instead of covering complete regions.…”

Section: Introductionmentioning

confidence: 99%

“…To explicitly model the scoring of proposal generation module rather than rely on attention weights without extra supervision (Mithun, Paul, and Roy-Chowdhury 2019), we are supposed to further measure the semantic similarity between the selected proposals and query for scoring refinement. Inspired by the success of recent works about masked language model (Devlin et al 2019;Song et al 2019;Wang, Li, and Smola 2019), we design a novel semantic completion module that predicts the important words (e.g.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Weakly-Supervised Video Moment Retrieval via Semantic Completion Network

Lin

Zhao

Zhang

et al. 2020

AAAI

128

107

View full text Add to dashboard Cite

Video moment retrieval is to search the moment that is most relevant to the given natural language query. Existing methods are mostly trained in a fully-supervised setting, which requires the full annotations of temporal boundary for each query. However, manually labeling the annotations is actually time-consuming and expensive. In this paper, we propose a novel weakly-supervised moment retrieval framework requiring only coarse video-level annotations for training. Specifically, we devise a proposal generation module that aggregates the context information to generate and score all candidate proposals in one single pass. We then devise an algorithm that considers both exploitation and exploration to select top-K proposals. Next, we build a semantic completion module to measure the semantic similarity between the selected proposals and query, compute reward and provide feedbacks to the proposal generation module for scoring refinement. Experiments on the ActivityCaptions and Charades-STA demonstrate the effectiveness of our proposed method.

show abstract

Section: Compared Methodsmentioning

confidence: 99%