Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

Abstract

Existing visual question answering methods tend to capture the spurious correlations from visual and linguistic modalities, and fail to discover the true casual mechanism that facilitates reasoning truthfully based on the dominant visual evidence and the correct question intention. Additionally, the existing methods usually ignore the complex event-level understanding in multi-modal settings that requires a strong cognitive capability of causal inference to jointly model cross-modal event temporality, causality, and dynamics. In this work, we focus on event-level visual question answering from a new perspective, i.e., cross-modal causal relational reasoning, by introducing causal intervention methods to mitigate the spurious correlations and discover the true causal structures for the integration of visual and linguistic modalities. Specifically, we propose a novel event-level visual question answering framework named Cross-Modal Causal RelatIonal Reasoning (CMCIR), which consists of three essential components named causality-aware visual-linguistic reasoning module, spatial-temporal transformer, and visual-linguistic feature fusion module, to achieve robust casuality-aware visual-linguistic question answering. To uncover the causal structures for visual and linguistic modalities, the novel Causality-aware Visual-Linguistic Reasoning (CVLR) module is proposed to collaboratively disentangle the visual and linguistic spurious correlations via elaborately designed front-door and back-door causal intervention modules. To discover the fine-grained interactions between linguistic semantics and spatial-temporal representations, we build a novel Spatial-Temporal Transformer (STT) that builds the multi-modal co-occurrence interactions between visual and linguistic content. To adaptively fuse the causality-ware visual and linguistic features, we introduce a Visual-Linguistic Feature Fusion (VLFF) module that leverages the hierarchical linguistic semantic relations as the guidance to learn the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on large-scale event-level urban dataset SUTD-TrafficQA and three benchmark real-world datasets TGIF-QA, MSVD-QA, and MSRVTT-QA demonstrate the effectiveness of our CMCIR for discovering visual-linguistic causal structures and achieving robust event-level visual question answering.

Semantic-Aware Dynamic Retrospective-Prospective Reasoning for Event-Level Video Question Answering

“…The experimental results on the test set of Traf-ficQA are shown in Table 1, where we also in-3 https://openai.com/blog/clip/ clude the previous baseline models for EVQA. 4 The results show that our proposed approach obtains accuracy of 43.19 under the multiple-choice setting, which surpasses previous state-of-the-art approaches including Eclipse , ERM , TMBC and CMCIR (Liu et al, 2022) by at least 4.5 points. Furthermore, our approach achieves an accuracy of 71.63 under Setting 1/2, outperforming previous strong baselines by at least 6 points.…”

“…) where 𭟋 is the function measuring the similarity between answer candidate and h T −1 , and y i,k represents the answer label for the i−th example -if the correct answer for the i−th example is the k−th answer then y i,k is 1 otherwise it is 0. Models Setting-1/4 Setting-1/2 Q-type (random) 25.00 50.00 QE-LSTM 25.21 50.45 QA-LSTM 26.65 51.02 Avgpooling 30.45 57.50 CNN+LSTM 30.78 57.64 I3D+LSTM 33.21 54.67 VIS+LSTM (Ren et al, 2015) 29.91 54.25 BERT-VQA (Yang et al, 2020) 33.68 63.50 TVQA (Lei et al, 2018) 35.16 63.15 HCRN (Le et al, 2020a) 36.49 63.79 Eclipse 37.05 64.77 ERM 37.11 65.14 TMBC 37.17 65.14 CMCIR (Liu et al, 2022) 38.58 N/A Ours 43.19 71.63…”

“…In general, the objective of the VQA task is to provide an answer to a visual-related question according to the content of an accompanying video. Despite significant recent progress in VQA, EVQA still remains one of the most challenging VQAbased tasks since it requires complex reasoning over the events across video frames (Sadhu et al, 2021;Zhong et al, 2022;Liu et al, 2022). To * *corresponding author tackle the challenges in EVQA, a number of approaches have been proposed .…”

“…propose a temporal-aware bidirectional attention mechanism for improving event reasoning in videos, while propose a novel model named Energy-based Refinedattention Mechanism (ERM), which obtains better performance compared to previous approaches with a smaller model size. Liu et al (2022), on the other hand, incorporate visual-linguistic causal dependencies based on Graph Convolutional Networks (Kipf and Welling, 2017) for enhancing cross-modal event reasoning for EVQA.…”

Urban Regional Function Guided Traffic Flow Prediction

“…Traffic flow prediction is a classic spatial-temporal prediction problem [29,24]. Recently, the mainstream approach is combining GCN [10] and RNN [16] to model spatial-temporal correlation [27,30,9,6,31,23,46], such as DCRNN [20], STGCN [48].…”