Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Abstract

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and topdown attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

Learning to Relate from Captions and Bounding Boxes

“…In this work, we propose a novel task of weaklysupervised relation prediction, with the objective of detecting relations between entities in an image purely from captions and object-level bounding box annotations without class information. Our proposed method builds upon top-down attention (Anderson et al, 2018), which generates captions and grounds word in these captions to entities in images.…”

“…Captioning using visual attention has proven to be very successful in aligning the words in a caption to their corresponding visual features, such as in Anderson et al (2018). As shown in Figure 1, we adopt the two-layer LSTM architecture in Anderson et al (2018); our end goal, however, is to associate each word with the closest object feature rather than producing a caption. The lower Attention LSTM cell takes in the words and the global image context vector (f , the mean of all features F), and its hidden state h a t acts as a query vector.…”

Multi-Layer Content Interaction Through Quaternion Product for Visual Question Answering

“…For each image I, we extract 100 region proposals and associated region features. However, different from bottom-up & top-down attention [25], We select the R ∈ R u×2×2×2048 image region feature as input. We map the dynamically changing question vector to the scaling factor and bias term of the channel feature through the fully connected layer fc and hc .…”

Image classification and captioning model considering a CAM‐based disagreement loss

“…The encoder-decoder model first extracts high-level visual features from a CNN trained on the image classification task, and then feeds the visual features into an RNN model to predict subsequent words of a caption for a given image. In recent years, a variety of successive models [2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][18][19][20] have achieved promising results. Semantic concept analysis, or attribute prediction [17,21], is a task closely related to image captioning, because attributes can be interpreted as a basis for descriptions.…”