Localizing Moments in Video with Temporal Language

Abstract

Localizing moments in a longer video via natural language queries is a new, challenging task at the intersection of language and video understanding. Though moment localization with natural language is similar to other language and vision tasks like natural language object retrieval in images, moment localization offers an interesting opportunity to model temporal dependencies and reasoning in text.We propose a new model that explicitly reasons about different temporal segments in a video, and shows that temporal context is important for localizing phrases which include temporal language. To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset. Our dataset consists of two parts: a dataset with real videos and template sentences (TEMPO -Template Language) which allows for controlled studies on temporal language, and a human language dataset which consists of temporal sentences annotated by humans (TEMPO -Human Language).

Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

“…Early works study this task in constrained settings, including the fixed spatial prepositions [21,38], instruction videos [1,31,35] and ordering constraint [4,37]. Recently, unconstrained query-based moment retrieval has attracted a lot of attention [6,10,13,14,22,23,42]. These methods are mainly based on a sliding window framework, which first samples candidate moments and then ranks these moments.…”

“…These methods are mainly based on a sliding window framework, which first samples candidate moments and then ranks these moments. Hendricks et al [13] propose a moment context network to integrate global and local video features for natural language retrieval, and the subsequent work [14] considers the temporal language by explicitly modeling the context structure of videos. Gao et al [10] develop a cross-modal temporal regression localizer to estimate the alignment scores of candidate moments and textual query, and then adjust the boundaries of high-score moments.…”

“…Most existing moment retrieval works [6,10,13,14,22,23,42] only focus on one aspect of this emerging task, such as the query representation learning [23], video context modeling [10,22] and cross-modal fusion [6,42], thus fail to develop a comprehensive system to further improve the performance of query-based moment retrieval. In this paper, we consider multiple crucial factors for highquality moment retrieval.…”

“…Thirdly, query-based moment retrieval requires the comprehensive reasoning of video and query contents, thus the crossmodal interaction is necessary for high-quality retrieval. Early approaches [10,13,14] ignore this factor and only simply combine the query and moment features for correlation estimations. Although recent methods [6,22,23,42] have developed a cross-modal interaction by widely-used attention mechanism, they still remain in the rough one-stage interaction, for example, highlighting the crucial context information of moments by the guidance of queries [22].…”

WSLLN:Weakly Supervised Natural Language Localization Networks

“…The goal is to retrieve a temporal segment from an untrimmed video based on an arbitrary text query. Recent work focuses on learning the mapping from visual segments to the input text (Hendricks et al, 2017;Gao et al, 2017a;Liu et al, 2018;Hendricks et al, 2018; Work done when the author was at Salesforce Research. † Corresponding author.…”