VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval

Ma, Minuk; Yoon, Sunjae; Kim, Junyeong; Lee, Young Joon; Kang, Sunghun; Yoo, Chang D.

doi:10.1007/978-3-030-58604-1_10

Cited by 61 publications

(53 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…And comparing to VLANet, FSAN performs overall better, except for R@5. This may be due to the surrogate proposal selection module introduced in VLANet (Ma et al, 2020), which in fact performs a two-stage candidate selection and gets rid of temporally overlapped candidates.…”

Section: Comparisons With State-of-the-art Methodsmentioning

confidence: 99%

Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding

Wang¹,

Zhou²,

Li³

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

Temporal language grounding (TLG) aims to localize a video segment in an untrimmed video based on a natural language description. To alleviate the expensive cost of manual annotations for temporal boundary labels, we are dedicated to the weakly supervised setting, where only video-level descriptions are provided for training. Most of the existing weakly supervised methods generate a candidate segment set and learn cross-modal alignment through a MIL-based framework. However, the temporal structure of the video as well as the complicated semantics in the sentence are lost during the learning. In this work, we propose a novel candidatefree framework: Fine-grained Semantic Alignment Network (FSAN), for weakly supervised TLG. Instead of view the sentence and candidate moments as a whole, FSAN learns token-by-clip cross-modal semantic alignment by an iterative cross-modal interaction module, generates a fine-grained cross-modal semantic alignment map, and performs grounding directly on top of the map. Extensive experiments are conducted on two widelyused benchmarks: ActivityNet-Captions, and DiDeMo, where our FSAN achieves state-ofthe-art performance.

show abstract

Section: Comparisons With State-of-the-art Methodsmentioning

confidence: 99%

Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding

Wang¹,

Zhou²,

Li³

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

show abstract

“…In the image domain, some works have dealt with image caption [43], image grounding [21], and text-to-image synthesis [6]. In the video domain, some works focus on temporal localization using natural language [32,35,51], where the temporal boundary needs to be localized with a given natural language description. The task of object tracking with natural language specification is similar, and the difference is to estimate the location of the interesting object marked with a bounding box.…”

Section: Textual and Visual Understandingmentioning

confidence: 99%

Capsule-based Object Tracking with Natural Language Specification

2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Tracking with Natural-Language Specification (TNL) is a joint topic of understanding the vision and natural language with a wide range of applications. In previous works, the communication between two heterogeneous features of vision and language is mainly through a simple dynamic convolution. However, the performance of prior works is capped by the difficulty of linguistic variation of natural language in modeling the dynamically changing target and its surroundings. In the meanwhile, natural language and vision are firstly fused and then utilized for tracking, which is hard to model the query-focused context. Query-focused should pay more attention to context modeling to promote the correlation between these two features. To address these issues, we propose a capsule-based network, referred to as CapsuleTNL, which performs regression tracking with natural language query. In the beginning, the visual and textual input is encoded with capsules, which can not only establish the relationship between entities but also the relationship between the parts of the entity itself. Then, we devise two interaction routing modules, which consist of visual-textual routing module to reduce the linguistic variation of input query and textual-visual routing module to precisely incorporate query-based visual cues simultaneously. To validate the potential of the proposed network for visual object tracking, we evaluate our method on two large tracking benchmarks. The experimental evaluation demonstrates the effectiveness of our capsule-based network.

show abstract

“…Mithun et al [2] first proposed weakly-supervised framework which performs SVMR without boundary annotations. The methods in [9,10] proposed to learn video-language alignment by embedding semantics into joint space. Although revolutionary ways of using weak supervision have been proposed, they do not fully explore weakly-supervised manner in a video corpus level.…”

Section: Video Moment Retrievalmentioning

confidence: 99%

“…As, in weakly-supervised manner, learning from boundary information is not available, generating multi-scale proposals and finding the most perti-nent one are important. Ma et al [10] contributes to selecting surrogate proposals in early stage. Although previous works [9,11] proposed efficient proposal selection, reducing redundant proposals and generating ones are still challenging.…”

Section: Video Proposal Generationmentioning

confidence: 99%

Weakly-Supervised Moment Retrieval Network for Video Corpus Moment Retrieval

Yoon

Kim

Hong³

et al. 2021

2021 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

This paper proposes Weakly-supervised Moment Retrieval Network (WMRN) for Video Corpus Moment Retrieval (VCMR), which retrieves pertinent temporal moments related to natural language query in a large video corpus. Previous methods for VCMR require full supervision of temporal boundary information for training, which involves a labor-intensive process of annotating the boundaries in a large number of videos. To leverage this, the proposed WMRN performs VCMR in a weakly-supervised manner, where WMRN is learned without ground-truth labels but only with video and text queries. For weakly-supervised VCMR, WMRN addresses the following two limitations of prior methods: (1) Blurry attention over video features due to redundant video candidate proposals generation, (2) Insufficient learning due to weak supervision only with video-query pairs. To this end, WMRN is based on (1) Text Guided Proposal Generation (TGPG) that effectively generates text guided multi-scale video proposals in the prospective region related to query, and (2) Hard Negative Proposal Sampling (HNPS) that enhances video-language alignment via extracting negative video proposals in positive video sample for contrastive learning. Experimental results show that WMRN achieves state-of-the-art performance on TVR and DiDeMo benchmarks in the weakly-supervised setting. To validate the attainments of proposed components of WMRN, comprehensive ablation studies and qualitative analysis are conducted.

show abstract

VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval

Cited by 61 publications

References 21 publications

Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding

Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding

Capsule-based Object Tracking with Natural Language Specification

Weakly-Supervised Moment Retrieval Network for Video Corpus Moment Retrieval

Contact Info

Product

Resources

About