Toward Remote Sensing Image Retrieval Under a Deep Image Captioning Perspective

Abstract

The performance of remote sensing image retrieval (RSIR) systems depends on the capability of the extracted features in characterizing the semantic content of images. Existing RSIR systems describe images by visual descriptors that model the primitives (such as different land-cover classes) present in the images. However, the visual descriptors may not be sufficient to describe the high-level complex content of RS images (e.g., attributes and relationships among different land-cover classes). To address this issue, in this paper we present a RSIR system that aims at generating and exploiting textual descriptions to accurately describe the relationships between the objects and their attributes present in RS images with captions (i.e., sentences). To this end, the proposed retrieval system consists of three main steps. The first step aims to encode the image visual features and then translate the encoded features into a textual description that summarizes the content of the image with captions. This is achieved based on the combination of a convolutional neural network with a recurrent neural network. The second step aims to convert the generated textual descriptions into semantically meaningful feature vectors. This is achieved by using the recent word embedding techniques. Finally, the last step estimates the similarity between the vectors of the textual descriptions of the query image and those of the archive images, and then retrieve the most similar images to the query image. Experimental results obtained on two different datasets show that the description of the image content with captions in the framework of RSIR leads to an accurate retrieval performance.

Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval

“…Although the retrieval method of generating caption solves human resources annotation, it may still leave some retrieval shortcomings. On the one hand, two-stage retrieval mode makes it difficult to avoid the loss of abundant information in the middle stage [13]. On the other hand, coarse caption generated by machine may not be used as a good representation of the RS image [14].…”

Semantic Annotation of Land Cover Remote Sensing Images Using Fuzzy CNN

“…The second module is devoted to semantic analysis. The semantic analysis module's primary function is to provide a higher-level knowledge of the given scenery and to annotate the given image with various class labels [14,15]. The following module is for intelligent feature extraction.…”

Modeling of Hyperparameter Tuned Deep Learning Model for Automated Image Captioning

“…This method is capable of generating proper captions, but it is not able to create semantically correct and imagespecific captions. In contrast to this, the novel image caption generation approach produces image caption from the visual contents using language models and then examines the visual content of an image [6]. In comparison with the above two classes, novel caption generation could create proper captions for the provided image, which is semantically correct when compared to earlier methods [7].…”