Interventional Video Grounding with Dual Contrastive Learning

Nan, Guoshun; Qiao, Rui; Xiao, Yao; Li, Jun; Leng, Sicong; Zhang, Hao; Lu, Wei

doi:10.1109/cvpr46437.2021.00279

Cited by 128 publications

(50 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, SeqPAN [110] introduces the concepts of named entity recognition [122]- [124] in NLP by splitting snippet sequence into begin, inside, and end regions of target moment, and background region. IVG-DCL [112] introduces dual contrastive learning mechanism to enhance multimodal interaction and leverages structured causal model [125] to address the selection bias of TSGV. CI-MHA [114] proposes to remedy the start/end prediction noise caused by annotator disagreement via an auxiliary moment segmentation task.…”

Section: Span-based Methodsmentioning

confidence: 99%

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

Zhang¹,

Sun²,

Wei³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Temporal sentence grounding in videos (TSGV), a.k.a., natural language video localization (NLVL) or video moment retrieval (VMR), aims to retrieve a temporal moment that semantically corresponds to a language query from an untrimmed video. Connecting computer vision and natural language, TSGV has drawn significant attention from researchers in both communities. This survey attempts to provide a summary of fundamental concepts in TSGV and current research status, as well as future research directions. As the background, we present a common structure of functional components in TSGV, in a tutorial style: from feature extraction from raw video and language query, to answer prediction of the target moment. Then we review the techniques for multimodal understanding and interaction, which is the key focus of TSGV for effective alignment between the two modalities. We construct a taxonomy of TSGV techniques and elaborate methods in different categories with their strengths and weaknesses. Lastly, we discuss issues with the current TSGV research and share our insights about promising research directions.

show abstract

Section: Span-based Methodsmentioning

confidence: 99%

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

Zhang¹,

Sun²,

Wei³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To remove the harmful confounding effects, we develop a deconfounding method, DCM, that first disentangles moment representation to learn the core feature of visual content and then intervenes the multimodal input based on backdoor adjustment. This is the first causality-based work that addresses the temporal location biases of VMR, which is significantly different from a new VMR work [21].…”

Section: Related Workmentioning

confidence: 96%

Deconfounded Video Moment Retrieval with Causal Intervention

Yang

Feng

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

150

View full text Add to dashboard Cite

We tackle the task of video moment retrieval (VMR), which aims to localize a specific moment in a video according to a textual query. Existing methods primarily model the matching relationship between query and moment by complex cross-modal interactions. Despite their effectiveness, current models mostly exploit dataset biases while ignoring the video content, thus leading to poor generalizability. We argue that the issue is caused by the hidden confounder in VMR, i.e., temporal location of moments, that spuriously correlates the model input and prediction. How to design robust matching models against the temporal location biases is crucial but, as far as we know, has not been studied yet for VMR.To fill the research gap, we propose a causality-inspired VMR framework that builds structural causal model to capture the true effect of query and video content on the prediction. Specifically, we develop a Deconfounded Cross-modal Matching (DCM) method to remove the confounding effects of moment location. It first disentangles moment representation to infer the core feature of visual content, and then applies causal intervention on the disentangled multimodal input based on backdoor adjustment, which forces the model to fairly incorporate each possible location of the target into consideration. Extensive experiments clearly show that our approach can achieve significant improvement over the state-of-theart methods in terms of both accuracy and generalization (Codes: https://github.com/Xun-Yang/Causal_Video_Moment_Retrieval). CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval.

show abstract

“…These works utilize data augmentation strategies to construct semantic related example pairs based on original data (e.g., random cropping and rotation of images [3]). Besides, contrastive learning has also been adopted to fuse multi-view information, such as image and text [19,29,33], text and graph [9]. It drives the representations of related information in different types to be similar, such that each kind of representations can be enhanced by each other.…”

Section: Contrastive Learningmentioning

confidence: 99%

C2-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender System

Zhou¹,

Zhou²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Conversational recommender systems (CRS) aim to recommend suitable items to users through natural language conversations. For developing effective CRSs, a major technical issue is how to accurately infer user preference from very limited conversation context. To address issue, a promising solution is to incorporate external data for enriching the context information. However, prior studies mainly focus on designing fusion models tailored for some specific type of external data, which is not general to model and utilize multi-type external data.To effectively leverage multi-type external data, we propose a novel coarse-to-fine contrastive learning framework to improve data semantic fusion for CRS. In our approach, we first extract and represent multi-grained semantic units from different data signals, and then align the associated multi-type semantic units in a coarse-to-fine way. To implement this framework, we design both coarse-grained and fine-grained procedures for modeling user preference, where the former focuses on more general, coarsegrained semantic fusion and the latter focuses on more specific, fine-grained semantic fusion. Such an approach can be extended to incorporate more kinds of external data. Extensive experiments on two public CRS datasets have demonstrated the effectiveness of our approach in both recommendation and conversation tasks. CCS CONCEPTS• Information systems → Users and interactive retrieval; Recommender systems.

show abstract

Interventional Video Grounding with Dual Contrastive Learning

Cited by 128 publications

References 63 publications

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

Deconfounded Video Moment Retrieval with Causal Intervention

C2-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender System

Contact Info

Product

Resources

About