Frame-Wise Cross-Modal Matching for Video Moment Retrieval

Query-graph with Cross-gating Attention Model for Text-to-Audio Grounding

“…Given a sentence query, their target is to localize an image region or a video moment in an image or video, respectively. Obviously, modeling pairwise relations between words in queries and capturing cross-modal interactions are also important for those tasks, so the attention mechanism [23], [25], [26] and the graph neural networks [27], [28], [29] are also adopted in some visual grounding methods. For example, Chen et al [30] proposed to explore the cross-modal interactions between the query and video by a Match-LSTM structure for temporal language grounding task; Liu et al proposed a ROLE model [25] which employed the query attention module to adaptively reweight the features of each word in query according to the video content.…”

Language-guided visual retrieval

“…To overcome the drawback of the anchor-based method, some anchor-free schemes are proposed. These methods [1,[46][47][48][49][50] usually organize a video as a continuous sequence holistically and process it with the sequence network like [8,51] to capture the temporal dependency. Then the interaction between the visual and linguistic sequences is usually applied in different attention operations.…”

“…For example, [46,47] first perform a binary classification for each frame in the sequence, then densely regress the distances from boundaries for all positive frames. And [1,[48][49][50] directly predict three probability scores for each frame being the foreground annotation, the start, and end boundaries with two kinds of classification losses. These methods greatly reduce the redundant computation bringing by candidate clips, and at the same time make good use of the temporal dependence in the video, which achieves appealing performance.…”

“…For prediction, the common practice is to fuse the linguistic feature with the clip's feature to get a fused representation which is then used for predicting an alignment score indicating the matching degree and the fine-tuning offset to the actual boundaries. On the other hand, anchor-free methods like [1,[46][47][48][49][50] model the video as a continuous sequence holistically and predict the specific start and end boundaries in the sequence. As for the detailed operations of prediction, [46,47] first perform a binary classification for each frame in the sequence, and then densely regress the distances from boundaries for all positive frames.…”

“…However, compared with the classification, it is difficult to regress the accurate boundary values directly with the local representations. Therefore, recently [1,[48][49][50] directly predict three probability scores for each frame being the foreground annotation, the start, and end boundaries with two kinds of classification loss. We also model the localization as a classification task, but the difference from previous jobs is that we only conduct the binary classification of every atomic representation in the sequence being the foreground or background and then localize the start and end boundaries in the obtained score vector through some postprocessing methods.…”