Video Graph Transformer for Video Question Answering

Abstract

We propose to perform video question answering (VideoQA) in a Contrastive manner via a Video Graph Transformer model (CoVGT). CoVGT's uniqueness and superiority are three-fold: 1) It proposes a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations and dynamics, for complex spatio-temporal reasoning. 2) It designs separate video and text transformers for contrastive learning between the video and text to perform QA, instead of multi-modal transformer for answer classification. Fine-grained video-text communication is done by additional cross-modal interaction modules. 3) It is optimized by the joint fully-and self-supervised contrastive objectives between the correct and incorrect answers, as well as the relevant and irrelevant questions respectively. With superior video encoding and QA solution, we show that CoVGT can achieve much better performances than previous arts on video reasoning tasks. Its performances even surpass those models that are pretrained with millions of external data. We further show that CoVGT can also benefit from cross-modal pretraining, yet with orders of magnitude smaller data. The results demonstrate the effectiveness and superiority of CoVGT, and additionally reveal its potential for more data-efficient pretraining. We hope our success can advance VideoQA beyond coarse recognition/description towards fine-grained relation reasoning of video contents. Our code will be available at https://github.com/doc-doc/CoVGT.

ATM: Action Temporality Modeling for Video Question Answering

“…Those methods that are only capable of descriptive content recognition cannot perform well, because they hardly capture the subtle transitions in the same scene in long-horizon. To this end, recent work [53,54] proposes to encode video as a local-to-global dynamic graph of spatiotemporal objects, so that the interaction relations can be encoded. However, the VideoQA models [17,53,54] built upon the dynamic graph of patches may easily be distracted by the object's appearance and capture limited motion information.…”

“…To this end, recent work [53,54] proposes to encode video as a local-to-global dynamic graph of spatiotemporal objects, so that the interaction relations can be encoded. However, the VideoQA models [17,53,54] built upon the dynamic graph of patches may easily be distracted by the object's appearance and capture limited motion information. We alleviate the distraction by a novel two-stage training to ensure a faithful representation of motions that are critical for temporality reasoning.…”

“…Concurrent work HiTEA [60] also introduces temporal shuffling, but the shuffling is used as evaluation test to metric the temporal reliance of datasets and VLP models, while our method leverages this in training a temporal reliable VideoQA. Concurrent work CoVGT [55] also investigates the contrastive learning in VideoQA. But as our action-centric contrastive learning aims at learning a faithful action representation, our contrastive objectives are action phrase in question, different from CoVGT's question and QA pairs.…”

“…However, the temporality reasoning in videos remains a great challenge, since these VideoQAs are only capable of holistic recognition of static content in a video. Recent work attempts to solve this issue by (1) enhancing the video representation with fine-grained dynamics [53,54] and (2) answering by grounding to question-critical visual evidence [35,36]. But it is hard to achieve a precise grounding, without the ground-truth of temporal boundaries for training.…”

“…But it is hard to achieve a precise grounding, without the ground-truth of temporal boundaries for training. The stateof-the-art method VGT [54] proposes to model the atomic actions across frames from the spatio-temporal dynamics of objects. In this way, the fine-grained dynamics can be captured.…”