Adaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QA

Jin, Weike; Zhao, Zhou; Cao, Xiaochun; Zhu, Jieming; He, Xiuqiang; Zhuang, Yueting

doi:10.1109/tip.2021.3076556

Cited by 25 publications

(9 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 7 shows the comparison results on the MSRVTT-QA dataset. It can be observed that our CMCIR performs better than the best performing method ASTG [103], with the highest accuracy of 38.9%. For What, Who, and When question types, the CMCIR performs the best compared with all the previous state-of-the-art methods.…”

Section: Results On Other Benchmark Datasetsmentioning

confidence: 91%

See 1 more Smart Citation

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

Liu¹,

Li²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Existing visual question answering methods tend to capture the spurious correlations from visual and linguistic modalities, and fail to discover the true casual mechanism that facilitates reasoning truthfully based on the dominant visual evidence and the correct question intention. Additionally, the existing methods usually ignore the complex event-level understanding in multi-modal settings that requires a strong cognitive capability of causal inference to jointly model cross-modal event temporality, causality, and dynamics. In this work, we focus on event-level visual question answering from a new perspective, i.e., cross-modal causal relational reasoning, by introducing causal intervention methods to mitigate the spurious correlations and discover the true causal structures for the integration of visual and linguistic modalities. Specifically, we propose a novel event-level visual question answering framework named Cross-Modal Causal RelatIonal Reasoning (CMCIR), which consists of three essential components named causality-aware visual-linguistic reasoning module, spatial-temporal transformer, and visual-linguistic feature fusion module, to achieve robust casuality-aware visual-linguistic question answering. To uncover the causal structures for visual and linguistic modalities, the novel Causality-aware Visual-Linguistic Reasoning (CVLR) module is proposed to collaboratively disentangle the visual and linguistic spurious correlations via elaborately designed front-door and back-door causal intervention modules. To discover the fine-grained interactions between linguistic semantics and spatial-temporal representations, we build a novel Spatial-Temporal Transformer (STT) that builds the multi-modal co-occurrence interactions between visual and linguistic content. To adaptively fuse the causality-ware visual and linguistic features, we introduce a Visual-Linguistic Feature Fusion (VLFF) module that leverages the hierarchical linguistic semantic relations as the guidance to learn the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on large-scale event-level urban dataset SUTD-TrafficQA and three benchmark real-world datasets TGIF-QA, MSVD-QA, and MSRVTT-QA demonstrate the effectiveness of our CMCIR for discovering visual-linguistic causal structures and achieving robust event-level visual question answering.

show abstract

Section: Results On Other Benchmark Datasetsmentioning

confidence: 91%

“…• ASTG [103]: A model that builds spatio-temporal relational graph to adaptively refine the temporal connections for dynamic object representations.…”

Section: Results On Other Benchmark Datasetsmentioning

confidence: 99%

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

Liu¹,

Li²,

Li³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[2] The aforementioned process mainly utilizes verbal and visual data and is vital for integrating them ingeniously. Currently, three typical visual-verbal integration tasks are: captioning, [3,4] visual question answering (VQA), [5][6][7][8][9] and visual dialog. [10] Image captioning is to generate descriptive text based on visual information.…”

Section: Introductionmentioning

confidence: 99%

“…information in both spatial and temporal spaces has already become a consensus in recent studies, [6][7][8]11] but attention seems not well explored because it degrades the performance, which is reported in ref. [6].…”

Section: Introductionmentioning

confidence: 99%

“…Attention has to be done both spatially and temporally as the target information may exist locally. Using both textual and visual information in both spatial and temporal spaces has already become a consensus in recent studies, [ 6–8,11 ] but attention seems not well explored because it degrades the performance, which is reported in ref. [6].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

TASTA: Text‐Assisted Spatial and Temporal Attention Network for Video Question Answering

Wang

Hou

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

Video question answering (VideoQA) is a typical task that integrates language and vision. The key for VideoQA is to extract relevant and effective visual information for answering a specific question. Information selection is believed to be necessary for this task due to the large amount of irrelevant information in the video, and explicitly learning an attention model can be a reasonable and effective solution for the selection. Herein, a novel VideoQA model called Text‐Assisted Spatial and Temporal Attention Network (TASTA) is proposed, which shows the great potential of explicitly modeling attention. TASTA is made to be simple, small, clean, and efficient for clear performance justification and possible easy extension. Its success is mainly from two new strategies of better using the textual information. Experimental results on a large and most representative dataset, TGIF‐QA, show the significant superiority of TASTA w.r.t. the state‐of‐the‐art and demonstrate the effectiveness of its key components via ablation studies.

show abstract

Advanced Models for Video Question Answering

Wu¹,

Wang²,

Wang³

et al. 2022

Visual Question Answering

View full text Add to dashboard Cite

Adaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QA

Cited by 25 publications

References 45 publications

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

TASTA: Text‐Assisted Spatial and Temporal Attention Network for Video Question Answering

Advanced Models for Video Question Answering

Contact Info

Product

Resources

About