Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

Abstract

Abstract. We address the problem of Visual Question Answering (VQA), which requires joint image and language understanding to answer a question about a given photograph. Recent approaches have applied deep image captioning methods based on convolutional-recurrent networks to this problem, but have failed to model spatial inference. To remedy this, we propose a model we call the Spatial Memory Network and apply it to the VQA task. Memory networks are recurrent neural networks with an explicit attention mechanism that selects certain parts of the information stored in memory. Our Spatial Memory Network stores neuron activations from different spatial regions of the image in its memory, and uses the question to choose relevant regions for computing the answer, a process of which constitutes a single "hop" in the network. We propose a novel spatial attention architecture that aligns words with image patches in the first hop, and obtain improved results by adding a second attention hop which considers the whole question to choose visual evidence based on the results of the first hop. To better understand the inference process learned by the network, we design synthetic questions that specifically require spatial inference and visualize the attention weights. We evaluate our model on two published visual question answering datasets, DAQUAR [1] and VQA [2], and obtain improved results compared to a strong deep baseline model (iBOWIMG) which concatenates image and question features to predict the answer [3].

WILDCAT: Weakly Supervised Learning of Deep ConvNets for Image Classification, Pointwise Localization and Segmentation

“…Similarly to WSL, the attention-based models [63,29,66,64] select relevant regions to support decisions. However the WSL methods usually include some structure on the selection process while it is implicit in attention-based approaches.…”

Multimodal Cross-guided Attention Networks for Visual Question Answering

“…Noh et al [15] adopted visual attention with joint loss minimization. Xu et al [16] obtained attention map by calculating the semantic similarity between image regions and the question. Ilievski et al [17] used an off-the-shelf object detector to catch the important regions, and then fed the regions into LSTM with global image features.…”

Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task

“…Our approach in the combination with the Normalized Correlation Analysis embedding technique improves on the state-of-the-art of the Visual Madlibs task. Text-Embedding Loss: Motivated by the popularity of deep architectures for visual question answering, that combine a global CNN image representation with an LSTM [7] question representation [4,13,17,20,29,30,31], as well as the leading performance of nCCA on the multi-choice Visual Madlibs task [32], we propose a novel extension of the CNN+LSTM architecture that chooses a prompt completion out of four candidates (see Figure 4) by measuring similarities directly in the embedding space. This contrasts with the prior approach of [32] that uses a post-hoc comparison between the discrete output of the CNN+LSTM method and all four candidates.…”

“…over the image, yield state of the art results [29,30,31]. Another, more focused "hard" attention, has also been studied in the image-to-text retrieval scenario [9] as well as fine-grained categorization [33], person recognition [19] and zero-shot learning [1].…”

“…We present our novel architecture that extends prior approaches on question answering about images [4,13,17,20,29,30,31] by learning similarity between candidate labels and internal output embedding of the neural network. Figure 3 depicts our architecture.…”