MarioQA: Answering Questions by Watching Gameplay Videos

Abstract

We present a framework to analyze various aspects of models for video question answering (VideoQA) using customizable synthetic datasets, which are constructed automatically from gameplay videos. Our work is motivated by the fact that existing models are often tested only on datasets that require excessively high-level reasoning or mostly contain instances accessible through single frame inferences. Hence, it is difficult to measure capacity and flexibility of trained models, and existing techniques often rely on adhoc implementations of deep neural networks without clear insight into datasets and models. We are particularly interested in understanding temporal relationships between video events to solve VideoQA problems; this is because reasoning temporal dependency is one of the most distinct components in videos from images. To address this objective, we automatically generate a customized synthetic VideoQA dataset using Super Mario Bros. gameplay videos so that it contains events with different levels of reasoning complexity. Using the dataset, we show that properly constructed datasets with events in various complexity levels are critical to learn effective models and improve overall performance.

Motion-Appearance Co-memory Networks for Video Question Answering

“…Comparing with images, temporal domain is unique to videos. A temporal attention mechanism is leveraged to selectively attend to one or more periods of a video in [16,24,35]. Besides temporal attention mecha- Figure 2.…”

IQA: Visual Question Answering in Interactive Environments

“…More recently, modular networks [4,20,25] that construct an explicit representation of the reasoning process by exploiting the compositional nature of language have been proposed. Similar architectures have also been applied to the video domain with extensions such as spatiotemporal attention [23,49]. Our proposed approach to question answering allows the agent to interact with its environment and is thus fundamentally different to past QA approaches.…”

Localizing Moments in Video with Temporal Language

“…In the video domain, the TGIF-QA (Jang et al, 2017) and Mario-QA (Mun et al, 2016) datasets provide opportunities to study temporal reasoning for the task of VQA. The TGIF-QA dataset considers three types of temporal questions: before/after questions, repetition count, and determining a repeating action.…”