Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding

Wang, Zhenzhi; Wang, Limin; Wu, Tao; Li, Tianhao; Wu, Guorong

doi:10.1609/aaai.v36i3.20163

Cited by 70 publications

(47 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the guidance of the generation of subquery-level information is not fully guaranteed. MMN [26] trains the model to distinguish matched and unmatched video-sentence pairs collected from intra-video and inter-video in order to find the relationship between positive and negative sentences. It also ignores the relationship between phrases and video.…”

Section: Fully-supervised Temporal Localizationmentioning

confidence: 99%

Phrase-level Prediction for Video Temporal Localization

Zheng

et al. 2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Video temporal localization aims to locate a period that semantically matches a natural language query in a given untrimmed video. We empirically observe that although existing approaches gain steady progress on sentence localization, the performance of phrase localization is far from satisfactory. In principle, the phrase should be easier to localize as fewer combinations of visual concepts need to be considered; such incapability indicates that the existing models only capture the sentence annotation bias in the benchmark but lack sufficient understanding of the intrinsic relationship between simple visual and language concepts, thus the model generalization and interpretability is questioned. This paper proposes a unified framework that can deal with both sentence and phrase-level localization, namely Phrase Level Prediction Net (PLPNet). Specifically, based on the hypothesis that similar phrases tend to focus on similar video cues, while dissimilar ones should not, we build a contrastive mechanism to restrain phrase-level localization without fine-grained phrase boundary annotation required in training. Moreover, considering the sentence's flexibility and wide discrepancy among phrases, we propose a clustering-based batch sampler to ensure that contrastive learning can be conducted efficiently. Extensive experiments demonstrate that our method surpasses state-of-the-art methods of phrase-level temporal localization while maintaining high performance in sentence localization and boosting the model's interpretability and generalization capability. Our code is available at https://github.com/sizhelee/PLPNet. CCS CONCEPTS• Computing methodologies → Visual content-based indexing and retrieval; Activity recognition and understanding.

show abstract

Section: Fully-supervised Temporal Localizationmentioning

confidence: 99%

Phrase-level Prediction for Video Temporal Localization

Zheng

et al. 2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

show abstract

“…Video grounding [2,11,14,29,42,45,52] is an important task in video understanding which aims to identify the timestamps semantically corresponding to the given query within the untrimmed videos. It remains a challenging task since it needs to not only model complex cross-modal interactions, but also capture comprehensive contextual information for semantic alignment.…”

Section: Introductionmentioning

confidence: 99%

“…(2) The semantic gap between DVC and video grounding datasets leads to errors between the generated dense caption and the ground truth. Specifically, We implement this data augmentation idea on two representative methods (i.e., MMN [42] and 2D-TAN [52]). The experimental results on ActivityNet Captions dataset are shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exploiting Prompt Caption for Video Grounding

Cao¹,

Cheng²,

Zhu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Video grounding aims to locate a moment of interest matching the given query sentence from an untrimmed video. Previous works ignore the sparsity dilemma in video annotations, which fails to provide the context information between potential events and query sentences in the dataset. In this paper, we contend that providing easily available captions which describe general actions i.e., templated captions defined in our paper, will significantly boost the performance. To this end, we propose a Templated Caption Network (TCNet) for video grounding. Specifically, we first introduce dense video captioning to generate dense captions, and then obtain templated captions by Non-Templated Caption Suppression (NTCS). To utilize templated captions better, we propose Caption Guided Attention (CGA) project the semantic relations between templated captions and query sentences into temporal space and fuse them into visual representations. Considering the gap between templated captions and ground truth, we propose Asymmetric Dual Matching Supervised Contrastive Learning (ADMSCL) for constructing more negative pairs to maximize cross-modal mutual information. Without bells and whistles, extensive experiments on three public datasets (i.e., ActivityNet Captions, TACoS and ActivityNet-CG) demonstrate that our method significantly outperforms state-of-the-art methods. Q1: A woman braids her hair slowly. Q2: She then ties the braids together and ties the end together. D4: The woman finishes and smiles at the end. D2: She then ties the braid on the side of the hair and ends by looking away. D1: She holds a bottle. D3: She takes out a rubber band.

show abstract

“…It also devises a novel multi-stage boundary regression to refine the predicted moments. Instead of using the simple Hadamard product, DMN [87] proposes to project proposals and query features to common embedding space and leverage metric learning for cross-modal pair discrimination.…”

Section: D-mapmentioning

confidence: 99%

Towards temporal sentence grounding in videos

Zhang¹

View full text Add to dashboard Cite

show abstract

Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding

Cited by 70 publications

References 78 publications

Phrase-level Prediction for Video Temporal Localization

Phrase-level Prediction for Video Temporal Localization

Exploiting Prompt Caption for Video Grounding

Towards temporal sentence grounding in videos

Contact Info

Product

Resources

About