Exploring Visual Relationship for Image Captioning

Abstract

It is always well believed that modeling relationships between objects would be helpful for representing and eventually describing an image. Nevertheless, there has not been evidence in support of the idea on image description generation. In this paper, we introduce a new design to explore the connections between objects for image captioning under the umbrella of attention-based encoder-decoder framework. Specifically, we present Graph Convolutional Networks plus Long Short-Term Memory (dubbed as GCN-LSTM) architecture that novelly integrates both semantic and spatial object relationships into image encoder. Technically, we build graphs over the detected objects in an image based on their spatial and semantic connections. The representations of each region proposed on objects are then refined by leveraging graph structure through GCN. With the learnt region-level features, our GCN-LSTM capitalizes on LSTM-based captioning framework with attention mechanism for sentence generation. Extensive experiments are conducted on COCO image captioning dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GCN-LSTM increases CIDEr-D performance from 120.1% to 128.7% on COCO testing set.

Attention on Attention for Image Captioning

“…We report the performance on the offline test split of our model as well as the compared models in Table 1. The models include: LSTM [37], which encodes the image using CNN and decodes it using LSTM; SCST [31], which employs a modified visual attention and is the first to use SCST to directly optimize the evaluation metrics; Up-Down [2], which employs a two-LSTM layer model with bottom-up features extracted from Faster-RCNN; RFNet [20], which fuses encoded features from multiple CNN networks; GCN-LSTM [49], which predicts visual relationships between every two entities in the image and encodes the relationship information into feature vectors; and SGAE [44], which introduces auto-encoding scene graphs into its model.…”

Hierarchy Parsing for Image Captioning

“…Since we design our HIP architecture to be a feature refiner or bank that outputs rich and multi-level representations of the image, it is feasible to plug HIP into any neural captioning models. We next discuss how to integrate hierarchy parsing into a general attention-based LSTM decoder in Up-Down [3] or a specific relation-augmented decoder in GCN-LSTM [34]. Please also note that our HIP is flexible to generalize to other vision tasks, e.g., recognition.…”

Relation Distillation Networks for Video Object Detection

“…Object relations characterize the interactions or geometric positions between objects. In the literature, there has been strong evidences on the use of object relation to support various vision tasks, e.g., recognition [48], object detection [17], cross-domain detection [2], and image captioning [52]. One representative work that employs object relation is [17] for object detection in images.…”