Multi-Turn Video Question Answering via Multi-Stream Hierarchical Attention Context Network

Abstract

Conversational video question answering is a challenging task in visual information retrieval, which generates the accurate answer from the referenced video contents according to the visual conversation context and given question. However, the existing visual question answering methods mainly tackle the problem of single-turn video question answering, which may be ineffectively applied for multiturn video question answering directly, due to the insufficiency of modeling the sequential conversation context. In this paper, we study the problem of multi-turn video question answering from the viewpoint of multi-step hierarchical attention context network learning. We first propose the hierarchical attention context network for context-aware question understanding by modeling the hierarchically sequential conversation context structure. We then develop the multi-stream spatio-temporal attention network for learning the joint representation of the dynamic video contents and context-aware question embedding. We next devise the hierarchical attention context network learning method with multi-step reasoning process for multi-turn video question answering. We construct two large-scale multi-turn video question answering datasets. The extensive experiments show the effectiveness of our method.

Video Dialog via Progressive Inference and Cross-Transformer

“…The existing visual dialog methods (Das et al, 2017a) mainly use recurrent neural network (like LSTM) to encode the dialog history as a single vector representation, which we think might be a bit rough and straightforward. Some more advanced methods (Seo et al, 2017;Zhao et al, 2018) utilize hierarchical structure, attention and memory mechanisms to refine the dialog history representation, which still lacks an explicit reasoning process. Recently, Kottur et al (2018) propose a neural module network architecture including two novel modules, which perform coreference resolution at a word level.…”

“…As for the task of video dialog, it's still less explored. One similar work is proposed by Zhao et al (2018). They study the problem of multi-turn video question answering by employing a hierarchical attention context learning method with recurrent neural networks for context-aware question understanding and a multi-stream attention network that learns the joint video representation.…”

“…Dataset. The datasets that we use are proposed in (Zhao et al, 2018), which are based on YouTube-Clips (Chen and Dolan, 2011) dataset and TACoS-MultiLevel (Rohrbach et al, 2014) dataset. These two datasets consist of 1,987 and 1,303 videos individually.…”

“…Besides the baseline models, we also compare our method with the HACRN model (Zhao et al, 2018). They propose a similar work called multiturn video question answering.…”

Multi-Question Learning for Visual Question Answering

“…Indeed, we observe that there are more and more VQA datasets (Jang et al 2017;Zhu et al 2016;Ranjay et al 2016) containing multiple questions for a video sequence. This is also a common phenomenon in applications such as education for children (Calhoun 1999) and multi-turn video question answering (Zhao et al 2018). It is worth noting that the semantic relational information among multiple video-question pairs plays an important role for human performing actually reasoning in VQA tasks.…”