Object Grounding via Iterative Context Reasoning

Activity-driven Weakly-Supervised Spatio-Temporal Grounding from Untrimmed Videos

“…Weakly-supervised Visual Grounding. Weakly-supervised image grounding [4,15,23] has been extended to video domain [5,7,14,25,35], but they are only applicable to constrained scenarios. Early work [33] grounded sentences to objects in the constrained videos that are recorded in lab.…”

“…Recently, great progress has been made on image grounding [4,15,30,31]. On the basis of this, researchers started to explore grounding in the video domain [5,7,14,25,35].…”

“…Nevertheless, in video grounding, it is labor-intensive to annotate a considerable number of bounding boxes for queries in videos. To address this challenge, multiple instance learning (MIL) methods [5,7,14,25,35] were proposed, which do not require bounding box annotations in the training videos. Video object grounding is achieved in a weakly-supervised fashion, where only a video and its description sentence are required during training.…”

“…This allows us to capture the physical states of the object as well as the spatial relations between human and the object. Most of existing visual grounding methods [5,15,25,35] simply compute the similarity between the visual and the textual features of the query object as a measurement for selecting candidate regions for the query. However, there is a granularity gap between the coarse textual and rich visual modalities.…”

“…Second, we propose an activity-driven region proposal refinement to find highquality region proposals. Most of existing visual grounding methods [5,15,25,35] build a candidate pool of top-region proposals, in which query objects could be missed. Proposals with a high recall rate by increasing typically lead to a large search space for grounding a query object.…”