Activity-driven Weakly-Supervised Spatio-Temporal Grounding from Untrimmed Videos

Abstract

In this paper, we study the problem of weakly-supervised spatiotemporal grounding from raw untrimmed video streams. Given a video and its descriptive sentence, spatio-temporal grounding aims at predicting the temporal occurrence and spatial locations of each query object across frames. Our goal is to learn a grounding model in a weakly-supervised fashion, without the supervision of both spatial bounding boxes and temporal occurrences during training. Existing methods have been addressed in trimmed videos, but their reliance on object tracking will easily fail due to frequent camera shot cut in untrimmed videos. To this end, we propose a novel spatio-temporal multiple instance learning framework for untrimmed video grounding. Spatial MIL and temporal MIL are mutually guided to ground each query to specific spatial regions and the occurring frames of a video. Furthermore, an activity described in the sentence is captured to use the informative contextual cues for region proposals refinement and text representation. We conduct extensive evaluation on YouCookII and RoboWatch datasets, and demonstrate our method outperforms state-of-the-art methods.

Weakly-Supervised Video Object Grounding via Stable Context Learning

“…Recently, Yang et al [41] simultaneously consider spatial and temporal contextual similarities of regions and frames in an end-to-end manner. Meanwhile, Chen et al [2] choose to enhance the textual representations of objects by exploiting the activities described in the sentence. While existing methods have not fully exploited the potential of the description sentences for vision-language alignment in rich contextual information and stable learning, we propose a novel frame-level MIL-based WSVOG framework which jointly enjoys the merits of them.…”

“…where Temp(•) is a function that synthetically considers the temporal region proposals and queried objects, and outputs the likelihood of each frame-bag being positive. There are multiple implementations of Temp(•) function in previous works [2,31,41], such as firstly concatenating the pooled features of region proposals and queried objects and then utilizing MLPs with softmax to transform them into likelihood [41].…”

“…We follow previous works [2,31,41] to evaluate the performance of our model using Box Accuracy and Query Accuracy. Box Accuracy is defined as the ratio of correctly grounded boxes to all of the grounded boxes.…”

“…In terms of textual representations, Zhou et al [49] employed the order of queried objects in the sentence as a language prior of when they appear in the video. Chen et al [2] exploited activity cues of the objects to enrich the textual representations. Despite their effectiveness, prior efforts have not fully exploited the potential of the description sentences for cross-modal alignment in two aspects:…”

“…Intuitively, contextual information is useful because it can provide detailed information of the object for accurate video object grounding. • A few works [2,31] have attempted to utilize contextualized object features (e.g. BERT features, GloVe word features after bi-LSTM), but found a significant decrease in performance.…”