A Survey on Temporal Sentence Grounding in Videos

Abstract

Temporal sentence grounding in videos (TSGV), which aims to localize one target segment from an untrimmed video with respect to a given sentence query, has drawn increasing attentions in the research community over the past few years. Different from the task of temporal action localization, TSGV is more flexible since it can locate complicated activities via natural languages, without restrictions from predefined action categories. Meanwhile, TSGV is more challenging since it requires both textual and visual understanding for semantic alignment between two modalities (i.e., text and video). In this survey, we give a comprehensive overview for TSGV, which i) summarizes the taxonomy of existing methods, ii) provides a detailed description of the evaluation protocols (i.e., datasets and metrics) to be used in TSGV, and iii) in-depth discusses potential problems of current benchmarking designs and research directions for further investigations. To the best of our knowledge, this is the first systematic survey on temporal sentence grounding. More specifically, we first discuss existing TSGV approaches by grouping them into four categories, i.e., two-stage methods, end-to-end methods, reinforcement learning-based methods, and weakly supervised methods. Then we present the benchmark datasets and evaluation metrics to assess current research progress. Finally, we discuss some limitations in TSGV through pointing out potential problems improperly resolved in the current evaluation protocols, which may push forwards more cutting edge research in TSGV. Besides, we also share our insights on several promising directions, including three typical tasks with new and practical settings based on TSGV.

MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions

“…Figure 1 shows that the current datasets comprise relatively short videos, containing single structured scenes, and language descriptions that cover most of the video. Furthermore, the temporal anchors for the language are temporally biased, leading to methods not learning from any visual features and eventually overfitting to temporal priors for specific actions, thus limiting their generalization capabilities [9,18].…”

MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions

“…Figure 1 shows that the current datasets comprise relatively short videos, contain single structured scenes, and language descriptions that cover most of the video. Furthermore, the temporal anchors for the language are temporally biased in time (refer to Figure 3), leading to methods not learning from any visual features and eventually overfitting to temporal priors for specific actions, thus limiting their generalization capabilities [7,16].…”