Hierarchical Visual-Textual Graph for Temporal Activity Localization via Language

Chen, Shaoxiang; Jiang, Yu‐Gang

doi:10.1007/978-3-030-58565-5_36

Cited by 40 publications

(32 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This section will compare our method with several state of the art methods. Since our model belongs to the one-stage methods, we mainly compare it with one-stage methods, which are ABLR [32], ExCL [9], DEBUG [18], TMLGA [24], HVTG [5], VSLnet [34], GDP [4], DRN [33], FIAN [22] and VLG-Net [21]. To further illustrate the effect, we also give the score of some two-stages methods including CTRL [8], SLTA [12], ACRN [16], CBP [26] and 2D-TAN [35].…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Local-enhanced Interaction for Temporal Moment Localization

Liang

Zhang

2021

Proceedings of the 2021 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Temporal moment localization via language aims to localize a video span in an untrimmed video which best matches the given natural language query. In most previous works, they try to match the whole query feature with multiple moment proposals, or match a global video embedding with phrase or word level query features. However, these coarse interaction models will become insufficient when the query-video contains more complex relationship. To address this issue, we propose a multi-branches interaction model for temporal moment localization. Specifically, the query sentence and video are encoded into multiple feature embeddings over several semantic sub-spaces. Then, each phrase embedding filters on a video feature to generate an attention sequence, which is used to reweight the video features. Moreover, a dynamic pointer decoder is developed to iteratively regress the temporal boundary, which can prevent our model from falling into a local optimum. To validate the proposed method, we have conducted extensive experiments on two popular benchmark datasets Charade-STA and TACoS. The experimental performance surpasses other state-of-the-arts methods, which demonstrates the effectiveness of our proposed model. CCS CONCEPTS• Information systems → Novelty in information retrieval.

show abstract

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Local-enhanced Interaction for Temporal Moment Localization

Liang

Zhang

2021

Proceedings of the 2021 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

show abstract

“…The majority of existing methods for video grounding can be categorized into two families: 1) proposal-based methods [2,5,13,14,17,25,26,33,43,45,48,50,54,56,57,58], which All codes and models will be made available shortly. generate a bunch of proposals in advance and select the best match with target spans, and 2) proposal-free methods [6,7,8,15,29,31,36,42,45,52,53,55], which estimate start and end timestamps aligned to the given description directly. The proposal-based approaches generally show strong performance at the expense of prohibitive cost of proposal generation.…”

Section: Introductionmentioning

confidence: 99%

Explore-And-Match: Bridging Proposal-Based and Proposal-Free With Transformer for Sentence Grounding in Videos

Woo¹,

Park²,

Inyong³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a new paradigm named explore-and-match for video grounding, which aims to seamlessly unify two streams of video grounding methods: proposal-based and proposal-free. To achieve this goal, we formulate video grounding as a set prediction problem and design an endto-end trainable Video Grounding Transformer (VIDGTR) that can utilize the architectural strengths of rich contextualization and parallel decoding for set prediction. The overall training is balanced by two key losses that play different roles, namely span localization loss and set guidance loss. These two losses force each proposal to regress the target timespan and identify the target query. Throughout the training, VIDGTR first explores the search space to diversify the initial proposals, and then matches the proposals to the corresponding targets to fit them in a fine-grained manner. The explore-and-match scheme successfully combines the strengths of two complementary methods, without encoding prior knowledge into the pipeline. As a result, VIDGTR sets new state-of-the-art results on two video grounding benchmarks with double the inference speed.

show abstract

“…Recently, [52] proposes obtaining contextualized activity representations in the video based on a language-conditioned message-passing algorithm where the activities are the composition of interactions between humans and objects in the scene. [53] also utilizes graph networks to model interactions between the objects and words as well as among the objects. These two jobs are similar to our model as they also utilize semantic features and devise graph operation in the network, but the organization and composition of their graphs are different from ours and they also fail to recognize the correspondence of objects in the time domain.…”

Section: Review Of Natural Language Video Localizationmentioning

confidence: 99%

“…While ExCL [48] and TMLGA [49] model the prediction of the span boundaries as the pure classification task. HVTG [53] and DORi [52] are the most similar work to ours as they also exploit the semantic feature of the scene. [53] utilizes graph networks to model the relationships of objects in the scene and the interactions between the objects and words.…”

Section: Experiments 431 Datasetsmentioning

confidence: 99%

“…HVTG [53] and DORi [52] are the most similar work to ours as they also exploit the semantic feature of the scene. [53] utilizes graph networks to model the relationships of objects in the scene and the interactions between the objects and words. The key idea of DORi is to learn a video feature embedding through a language-conditioned message-passing algorithm model.…”

Section: Experiments 431 Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Language-guided visual retrieval

Su¹

View full text Add to dashboard Cite

show abstract

Hierarchical Visual-Textual Graph for Temporal Activity Localization via Language

Cited by 40 publications

References 45 publications

Local-enhanced Interaction for Temporal Moment Localization

Local-enhanced Interaction for Temporal Moment Localization

Explore-And-Match: Bridging Proposal-Based and Proposal-Free With Transformer for Sentence Grounding in Videos

Language-guided visual retrieval

Contact Info

Product

Resources

About