Progressive Attention Memory Network for Movie Story Question Answering

Abstract

This paper proposes the progressive attention memory network (PAMN) for movie story question answering (QA). Movie story QA is challenging compared to VQA in two aspects: (1) pinpointing the temporal parts relevant to answer the question is difficult as the movies are typically longer than an hour, (2) it has both video and subtitle where different questions require different modality to infer the answer. To overcome these challenges, PAMN involves three main features: (1) progressive attention mechanism that utilizes cues from both question and answer to progressively prune out irrelevant temporal parts in memory, (2) dynamic modality fusion that adaptively determines the contribution of each modality for answering the current question, and (3) belief correction answering scheme that successively corrects the prediction score on each candidate answer. Experiments on publicly available benchmark datasets, MovieQA and TVQA, demonstrate that each feature contributes to our movie story QA architecture, PAMN, and improves performance to achieve the state-of-the-art result. Qualitative analysis by visualizing the inference mechanism of PAMN is also provided.

Dual Hierarchical Temporal Convolutional Network with QA-Aware Dynamic Normalization for Video Story Question Answering

“…denotes the frame number. Following [20,26], we also use detected object labels (referred to as visual concept) instead of ImageNet features as visual input on the TVQA dataset (see Table 1).…”

“…With the development of deep learning techniques (e.g., neural memory networks [15], attention mechanisms [28]), studies on QA have been rapidly advanced, and showed prominent performance improvement. In this paper, we focus on the video story QA task [20,23,36], which distinctively requires machines to understand the video contents and storylines based on temporally-aligned videos and subtitles, and thus answer multiple choice questions correctly. An example of video story QA can be seen in Figure 1.…”

“…Recent works [20,23,24,36,45] have introduced memory networks [39] plus attention mechanisms for video story QA. Memory networks enable a model to cache a large amount of sequential information in memory slots, and attention mechanisms are used to retrieve the most relevant information to a given query.…”

“…Second, most existing works model static and rough interactions between the multimodal inputs. For example, in [20,23,36,44], the proposed approaches simply aggregate the sentence representation by global mean/max pooling first and then combine the obtained vector with the video representation, such that the dynamic and fine-grained interactions between each word and each video feature unit are not captured. Third, previous methods do not fully exploit both question and answer choices (QA pairs) in each stage of the network.…”

“…Instead, they only use answer choices in final prediction stage to obtain the score of each candidate answer. Kim et al [20] developed Progressive Attention Memory Networks (PAMN) that separately utilize question and answer in different attention stages. These existing methods are limited by their insufficient (or shallow) exploitation of question and answer choices, producing suboptimal results.…”