A Read-Write Memory Network for Movie Story Understanding

Abstract

We propose a novel memory network model named Read-Write Memory Network (RWMN) to perform question and answering tasks for large-scale, multimodal movie story understanding. The key focus of our RWMN model is to design the read network and the write network that consist of multiple convolutional layers, which enable memory read and write operations to have high capacity and flexibility. While existing memory-augmented network models treat each memory slot as an independent block, our use of multi-layered CNNs allows the model to read and write sequential memory cells as chunks, which is more reasonable to represent a sequential story because adjacent memory blocks often have strong correlations. For evaluation, we apply our model to all the six tasks of the MovieQA benchmark [26], and achieve the best accuracies on several tasks, especially on the visual QA task. Our model shows a potential to better understand not only the content in the story, but also more abstract information, such as relationships between characters and the reasons for their actions.

Progressive Attention Memory Network for Movie Story Question Answering

“…While iteratively taking question and answers for temporal attention generation, memories are progressively updated to accumulate cues to locate relevant temporal parts for answering the question. We observed that single-step temporal attention on memory networks [26,21] often generates blurry attention map. The multi-step nature of progressive attention mechanism enables generating sharper attention distribution.…”

“…The attention mechanism have widely been adopted to retrieve the information relevant to the question. We observed that single-step attention on memory networks [21,26] often generates blurred temporal attention map.…”

“…In recent years, studies on question answering (QA) have successfully benefited from deep neural networks, and showed remarkable performance improvement on textQA [24,30], imageQA [2,3,19,31], videoQA [8,11,32,34]. This paper considers movie story QA [15,18,21,26,29] that aims at a joint understanding of vision and language by answering questions about movie contents and storyline after observing temporally-aligned video and subtitle. Movie * This research was supported by Samsung Research story QA is challenging compared to VQA in following two aspects: (1) pinpointing the temporal parts relevant to answer the question is difficult as the movies are typically longer than an hour and (2) it has both video and subtitle where different questions require different modality to infer the answer.…”

“…The information in the movie required to answer the question is not distributed uniformly across the temporal axis. To address this issue, memory networks [24] have widely been accepted in QA tasks [21,24,26,30]. The attention mechanism have widely been adopted to retrieve the information relevant to the question.…”

“…This observation is modeled by belief correction answering scheme. The prediction score (logits), which this paper refers to a belief, is equally likely initialized and successively corrected compared to existing answering scheme [15,21,29] which uses single-step answering scheme.…”