Multi-stage Aggregated Transformer Network for Temporal Language Localization in Videos

Zhang, Mingxing; Yang, Yang; Chen, Xinghan; Ji, Yanli; Xu, Xing; Li, Jingjing; Shen, Heng Tao

doi:10.1109/cvpr46437.2021.01248

Cited by 63 publications

(49 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [117], [142], SA is entirely replaced by CA, but it can also be kept [54]. Indeed, [53] has their modalities co-attending to each other and self-attending to themselves, claiming this keeps intra-modal and inter-modal dynamics separate, up to some degree. In contrast to encoder fusion, the computational cost is reduced to O((…”

Section: Cross-attention Fusion (Caf)mentioning

confidence: 99%

“…Other approaches involve discretizing video tokens using quantization [50] or classifying the contents present in them [13]. Also, in [53] authors propose an asymmetrical use of MTP where only textual tokens are masked, but visual context is used to reconstruct them. The most common approach, however, is found in other works that propose adapting a variant of the Noise Contrastive Estimation (NCE) [178], which we detail when discussing contrastive losses.…”

Section: Self-supervised Tasksmentioning

confidence: 99%

“…Text. When leveraging text (sequences of words) as an additional modality for VTs, the embeddings techniques commonly utilized are GloVe [215] (e.g., [53], [54], [55]), Word2Vec (e.g., [104]) or, directly, the representations provided by a pre-trained BERT [2] (e.g., [65], [70], [71], [104]). Compared to BERT, GloVe and Word2Vec are simpler methods based on, respectively, matrix factorization and shallow neural networks that are trained unsupervisedly based on statistical patterns, such as co-occurrence.…”

Section: S11 Text and Audio Embeddingmentioning

confidence: 99%

“…In practice, decoder-only models in vision are mostly used to autoregressively decode captions describing visual input data. Prompting video frames will mix two very different representations in the decoder, so CNNs substitute the encoder to provide context [51], [52], [53]. Both encoderonly and decoder-only layers consist of SA and FF sublayers interleaved with Add+Norm after each of them.…”

Section: Transformer Trends Adopted For Videomentioning

confidence: 99%

See 3 more Smart Citations

Video Transformers: A Survey

Selva¹,

Johansen²,

Escalera³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer models have shown great success modeling long-range interactions. Nevertheless, they scale quadratically with input length and lack inductive biases. These limitations can be further exacerbated when dealing with the high dimensionality of video. Proper modeling of video, which can span from seconds to hours, requires handling long-range interactions. This makes Transformers a promising tool for solving video related tasks, but some adaptations are required. While there are previous works that study the advances of Transformers for vision tasks, there is none that focus on in-depth analysis of video-specific designs. In this survey we analyse and summarize the main contributions and trends for adapting Transformers to model video data. Specifically, we delve into how videos are embedded and tokenized, finding a very widspread use of large CNN backbones to reduce dimensionality and a predominance of patches and frames as tokens. Furthermore, we study how the Transformer layer has been tweaked to handle longer sequences, generally by reducing the number of tokens in single attention operation. Also, we analyse the self-supervised losses used to train Video Transformers, which to date are mostly constrained to contrastive approaches. Finally, we explore how other modalities are integrated with video and conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D CNN counterparts with equivalent FLOPs and no significant parameter increase.

show abstract

Section: Cross-attention Fusion (Caf)mentioning

confidence: 99%

Section: Self-supervised Tasksmentioning

confidence: 99%

Section: S11 Text and Audio Embeddingmentioning

confidence: 99%

Section: Transformer Trends Adopted For Videomentioning

confidence: 99%

See 2 more Smart Citations

Video Transformers: A Survey

Selva¹,

Johansen²,

Escalera³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…SV-VMR [94] decomposes query into multiple semantic roles [95] and performs multi-level cross-modal reasoning at semantic level. MATN [52] further concatenates proposals and query words into a sequence, and encodes them through a single-stream transformer network. It also devises a novel multi-stage boundary regression to refine the predicted moments.…”

Section: Temporal Adjacent Networkmentioning

confidence: 99%

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

Zhang¹,

Sun²,

Wei³

et al. 2022

Preprint

View full text Add to dashboard Cite

Temporal sentence grounding in videos (TSGV), a.k.a., natural language video localization (NLVL) or video moment retrieval (VMR), aims to retrieve a temporal moment that semantically corresponds to a language query from an untrimmed video. Connecting computer vision and natural language, TSGV has drawn significant attention from researchers in both communities. This survey attempts to provide a summary of fundamental concepts in TSGV and current research status, as well as future research directions. As the background, we present a common structure of functional components in TSGV, in a tutorial style: from feature extraction from raw video and language query, to answer prediction of the target moment. Then we review the techniques for multimodal understanding and interaction, which is the key focus of TSGV for effective alignment between the two modalities. We construct a taxonomy of TSGV techniques and elaborate methods in different categories with their strengths and weaknesses. Lastly, we discuss issues with the current TSGV research and share our insights about promising research directions.

show abstract