Graph-Based Multi-Interaction Network for Video Question Answering

Gu, Mao; Zhao, Zhou; Jin, Weike; Hong, Richang; Wu, Fei

doi:10.1109/tip.2021.3051756

Cited by 34 publications

(6 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For What, Who, and When types, the CMCIR outperforms all the comparison methods significantly. Although GMIN [100] and CASSG [104] perform marginally better than our CMCIR for How and Where types, our CMCIR performs significantly better than GMIN for What (+8.3%), Who (+9.0%), When (+1.6%), and the overall (+8.3%) tasks.…”

Section: Results On Other Benchmark Datasetsmentioning

confidence: 68%

See 1 more Smart Citation

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

Liu¹,

Li²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Existing visual question answering methods tend to capture the spurious correlations from visual and linguistic modalities, and fail to discover the true casual mechanism that facilitates reasoning truthfully based on the dominant visual evidence and the correct question intention. Additionally, the existing methods usually ignore the complex event-level understanding in multi-modal settings that requires a strong cognitive capability of causal inference to jointly model cross-modal event temporality, causality, and dynamics. In this work, we focus on event-level visual question answering from a new perspective, i.e., cross-modal causal relational reasoning, by introducing causal intervention methods to mitigate the spurious correlations and discover the true causal structures for the integration of visual and linguistic modalities. Specifically, we propose a novel event-level visual question answering framework named Cross-Modal Causal RelatIonal Reasoning (CMCIR), which consists of three essential components named causality-aware visual-linguistic reasoning module, spatial-temporal transformer, and visual-linguistic feature fusion module, to achieve robust casuality-aware visual-linguistic question answering. To uncover the causal structures for visual and linguistic modalities, the novel Causality-aware Visual-Linguistic Reasoning (CVLR) module is proposed to collaboratively disentangle the visual and linguistic spurious correlations via elaborately designed front-door and back-door causal intervention modules. To discover the fine-grained interactions between linguistic semantics and spatial-temporal representations, we build a novel Spatial-Temporal Transformer (STT) that builds the multi-modal co-occurrence interactions between visual and linguistic content. To adaptively fuse the causality-ware visual and linguistic features, we introduce a Visual-Linguistic Feature Fusion (VLFF) module that leverages the hierarchical linguistic semantic relations as the guidance to learn the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on large-scale event-level urban dataset SUTD-TrafficQA and three benchmark real-world datasets TGIF-QA, MSVD-QA, and MSRVTT-QA demonstrate the effectiveness of our CMCIR for discovering visual-linguistic causal structures and achieving robust event-level visual question answering.

show abstract

Section: Results On Other Benchmark Datasetsmentioning

confidence: 68%

“…• GMIN [100]: A graph-based relation-aware neural network that explores the relationships and dependencies between objects spatially and temporally.…”

Section: Results On Other Benchmark Datasetsmentioning

confidence: 99%

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

Liu¹,

Li²,

Li³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the monolithic graph is cumbersome to extend to long videos with multiple objects. More recently, GMIN (Gu et al 2021) builds a spatio-temporal graph over object trajectories and shows improvements over its attention version (Jin et al 2019). While L-GCN and GMIN construct query-blind graphs, HGA (Jiang and Han 2020), DualVGR (Wang, Bao, and Xu 2021) and B2A (Park, Lee, and Sohn 2021) design query-specific graphs for better performance.…”

Section: Related Workmentioning

confidence: 99%

“…In Table 1 and Table 2, we compare our model with some established VideoQA techniques covering 4 major categories: 1) cross-attention (e.g., ST-VQA (Jang et al 2017), PSAC (Li et al 2019b), STA (Gao et al 2019), MIN (Jin et al 2019) and QueST )), 2) motion-appearance memory (e.g., AMU (Xu et al 2017), Co-Mem (Gao et al 2018) and HME (Fan et al 2019)), 3) graph-structured models (e.g., L-GCN 2 (Huang et al 2020), HGA (Jiang and Han 2020), DualVGR (Wang, Bao, and Xu 2021), GMIN (Gu et al 2021) and B2A (Park, Lee, and Sohn 2021)) and 4) hierarchical models (e.g., HCRN (Le et al 2020) and HOSTR (Dang et al 2021)). The results show that our Hierarchical QGA (HQGA) model performs consistently better than the others on all the experimented datasets.…”

Section: The State Of the Art Comparisonmentioning

confidence: 99%

Video as Conditional Graph Hierarchy for Multi-Granular Question Answering

Xiao

Yao

Liu

et al. 2022

AAAI

View full text Add to dashboard Cite

Video question answering requires the models to understand and reason about both the complex video and language data to correctly derive the answers. Existing efforts have been focused on designing sophisticated cross-modal interactions to fuse the information from two modalities, while encoding the video and question holistically as frame and word sequences. Despite their success, these methods are essentially revolving around the sequential nature of video- and question-contents, providing little insight to the problem of question-answering and lacking interpretability as well. In this work, we argue that while video is presented in frame sequence, the visual elements (e.g., objects, actions, activities and events) are not sequential but rather hierarchical in semantic space. To align with the multi-granular essence of linguistic concepts in language queries, we propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner, with the guidance of corresponding textual cues. Despite the simplicity, our extensive experiments demonstrate the superiority of such conditional hierarchical graph architecture, with clear performance improvements over prior methods and also better generalization across different type of questions. Further analyses also demonstrate the model's reliability as it shows meaningful visual-textual evidences for the predicted answers.

show abstract

“…New generalizations and definitions are developed to handle the complexity of structured data, so a graph convolution is generalized from a 2D convolution and GNNs are evolved to overcome the CNNs complexity [12]. GCN performs generalized convolutions from image to graph [13], and has been adopted in many applications efficiently [14], [15], [16]. Natural structure of the human skeleton is the geometric space graph, where the joints as nodes and their natural bones relations in the human body as edges.…”

Section: Introductionmentioning

confidence: 99%