Multi-Question Learning for Visual Question Answering

Abstract

Visual Question Answering (VQA) raises a great challenge for computer vision and natural language processing communities. Most of the existing approaches consider video-question pairs individually during training. However, we observe that there are usually multiple (either sequentially generated or not) questions for the target video in a VQA task, and the questions themselves have abundant semantic relations. To explore these relations, we propose a new paradigm for VQA termed Multi-Question Learning (MQL). Inspired by the multi-task learning, MQL learns from multiple questions jointly together with their corresponding answers for a target video sequence. The learned representations of video-question pairs are then more general to be transferred for new questions. We further propose an effective VQA framework and design a training procedure for MQL, where the specifically designed attention network models the relation between input video and corresponding questions, enabling multiple video-question pairs to be co-trained. Experimental results on public datasets show the favorable performance of the proposed MQL-VQA framework compared to state-of-the-arts.

Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training

“…CUPID [47] is a recent work studying the domain gap between video-language pre-training and fine-tuning. These pre-trained models have been applied in various downstream applications, e.g., action recognition [13], video caption [6,29], video retrieval [43], and video question answering [11,16]. The video-language data has more complex semantic structures and relationships than the image-language data, leading video-language pre-training to be a much more challenging research problem.…”

Recent Advances in Video Question Answering: A Review of Datasets and Methods

“…However,there could be many questions addressed to a particular video and most of them have abundant semantic relations. In order to explore these semantic relations, [26] proposes Multi-Question Learning (MQL) that learns multiple questions jointly with their candidate answers for a particular video sequence. These joint learned representations of video and question can then be used to learn new questions.…”

“…These joint learned representations of video and question can then be used to learn new questions. [26] introduces an efficient framework and training paradigm for MQL, where the relations between video and questions are modeled using attention network. This framework enables the co-training of multiple video-question pairs.…”

“…This framework enables the co-training of multiple video-question pairs. MQL [26] is capable to perform better understanding and reasoning even in the case of a single question. Authors carry out extensive experiments on 2 popular datasets-TVQA and CPT-QA.…”

VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks

“…Li et al (2019); Fan et al (2019); Jiang and Han (2020) model interaction between all pairs of question token-level representations and temporal-level features of the input video through similarity matrix, memory networks, and graph networks respectively. ; Le et al (2019cLe et al ( , 2020b; Lei et al (2020); Huang et al (2020) extends the previous approach by dividing a video into equal segments, sub-sampling video frames, or considering objectlevel representations of input video. We propose to replace token-level and global question representations with compositional question representations of specific entities and actions.…”