Dynamic Spatio-Temporal Modular Network for Video Question Answering

STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering

“…We compare with 3 representative video QA models: HME (Fan et al 2019) is a memory-network-based model to encode video and text features; HCRN (Le et al 2020) uses conditional relational networks to build a hierarchical structure that learns video representation on both clip level and video level; PSAC (Li et al 2019) uses both video and question positional self-attention instead of RNNs to model dependencies of questions and temporal relationships of videos. To compare with models that explicitly model the multi-step reasoning process, we also compare with DSTN (Qian et al 2022), a neural module network concurrent to our work, and MAC (Hudson and Manning 2018) which performs iterative attention-based reasoning with a recurrent "Memory, Attention and Composition" cell. We make minor modifications on the attention of MAC to attend to 2-D (T × dim V ) temporal features instead of 3-D (H × W × dim V ) spatial features.…”

“…Temporal module is designed to transform the attention scores according to the switch keyword s. We use the same metric IoU att to evaluate the attention scores output att out . Inspired by (Qian et al 2022), we randomly sample two frames as the start and end frames as baseline results. Specially, the start frame is always the first frame when s = 'before', and the end frame is always the last frame when s = 'after'.…”

“…Contrary to the intense research efforts of NMNs on image QA, there are significantly fewer works that focus on video QA (Le, Chen, and Hoi 2022;Qian et al 2022). Though sharing the same motivation, it is non-trivial to define the sub-tasks and design their corresponding modules for video modality, which is one of the main contributions of our work.…”

“…Though the above-mentioned methods have achieved outstanding performance, for most of these methods their performance tends to drop when evaluating on questions that require complicated logical reasoning or counterfactual questions and are difficult to interpret. To tackle these problems, some works use neural symbolic approach (Yi et al 2019) (Qian et al 2022) or construct physics models (Ding et al 2021;Chen et al 2021).…”

“…Though sharing the same motivation, it is non-trivial to define the sub-tasks and design their corresponding modules for video modality, which is one of the main contributions of our work. The work most similar to ours is DSTN (Qian et al 2022), which also uses neural module network for video QA. But our work is significantly different from theirs in better performance, better explainability, the usage of intermediate supervision, the ability to collaborate with pretrained models, and verifying its applicability when program annotations are not available.…”