A Multi-Level Attention Model for Remote Sensing Image Captions

Abstract

The task of image captioning involves the generation of a sentence that can describe an image appropriately, which is the intersection of computer vision and natural language. Although the research on remote sensing image captions has just started, it has great significance. The attention mechanism is inspired by the way humans think, which is widely used in remote sensing image caption tasks. However, the attention mechanism currently used in this task is mainly aimed at images, which is too simple to express such a complex task well. Therefore, in this paper, we propose a multi-level attention model, which is a closer imitation of attention mechanisms of human beings. This model contains three attention structures, which represent the attention to different areas of the image, the attention to different words, and the attention to vision and semantics. Experiments show that our model has achieved better results than before, which is currently state-of-the-art. In addition, the existing datasets for remote sensing image captioning contain a large number of errors. Therefore, in this paper, a lot of work has been done to modify the existing datasets in order to promote the research of remote sensing image captioning.

Using Neural Encoder-Decoder Models With Continuous Outputs for Remote Sensing Image Captioning

“…The current state-of-the-art results for remote sensing image captioning were reported by Li et al [3], leveraging a novel multi-level attention mechanism that uses three attention structures: one focusing on the different image regions; another focusing on the previously generated words; and a separate one deciding whether to attend to the image information or the caption information. In addition, these authors have used and released modified versions of the three aforementioned datasets, correcting a variety of errors in the textual descriptions associated to the images.…”

“…In our work, we also assessed model performance using UCM and RSCID, thus considering the larger dataset that is currently available in the area, as well as a smaller dataset that allows us to see how performance varies as a function of the available amount of training data. It should nonetheless be noted that the values in Table 1 are not all directly comparable, due to the fact that different studies used either the original or the updated versions of the datasets from Li et al [3], or due to small differences in the computation of the evaluation metrics (e.g., different estimates for the IDF term weight component in metrics such as CIDEr).…”

“…T HE idea of interacting with remote sensing imagery through natural language has been gaining increased interest [1], [2], [3]. Automatically annotating remote sensing images with short text descriptions (i.e., captions) can be an effective approach to describe the contents of large image repositories concerning specific areas of the Earth.…”

Learning to Hash with Convolutional Network for Multi-label Remote Sensing Image Retrieval

“…Residual attention [20] is integrated to train a stacked deep neural network to leverage the classification accuracy. Several attention-based image retrievals had been explored in different studies [18,[21][22][23][24][36][37][38][39].…”

“…Then hashing layer is appended to learn binary hash codes. Different loss function [18,[21][22][23][24][36][37][38][39] had been explored including point-wise similarity, pair-wise similarity [37,39] and triplet-wise supervision [38] to learn binary hash codes. In [26], different loss had been considered to train to generic feature.…”