Deep Unsupervised Embedding for Remote Sensing Image Retrieval Using Textual Cues

Abstract

Compared to image-image retrieval, text-image retrieval has been less investigated in the remote sensing community, possibly because of the complexity of appropriately tying textual data to respective visual representations. Moreover, a single image may be described via multiple sentences according to the perception of the human labeler and the structure/body of the language they use, which magnifies the complexity even further. In this paper, we propose an unsupervised method for text-image retrieval in remote sensing imagery. In the method, image representation is obtained via visual Big Transfer (BiT) Models, while textual descriptions are encoded via a bidirectional Long Short-Term Memory (Bi-LSTM) network. The training of the proposed retrieval architecture is optimized using an unsupervised embedding loss, which aims to make the features of an image closest to its corresponding textual description and different from other image features and vise-versa. To demonstrate the performance of the proposed architecture, experiments are performed on two datasets, obtaining plausible text/image retrieval outcomes.

Remote Sensing Cross-Modal Retrieval by Deep Image-Voice Hashing

“…1) Image-text retrieval method: Image-text retrieval method [15], [16] tries to search desired images (voices) according to the corresponding text description (images). Abdullah et al [15] proposed a Deep Bidirectional Triplet Network (DBTN) to fuse the multiple text description with an average fusion strategy.…”

“…Abdullah et al [15] proposed a Deep Bidirectional Triplet Network (DBTN) to fuse the multiple text description with an average fusion strategy. Rahhal et al [16] proposed a visual Big Transfer (BiT) Models and a Bidirectional Long Short-Term Memory (Bi-LSTM) network to learn image and text features, respectively. Cheng et al [32] proposed a Semantic Alignment Module (SAM) to enhance the latent correlation between remote sensing images and text features.…”

“…Abdullah et al [15] unified Long Short Term Memory Network (LSTM) and pre-trained Convolutional Neural Network (CNN) to design a deep bidirectional triplet network for remote sensing image-text retrieval. Rahhal et al [16] designed a novel unsupervised loss for unsupervised remote sensing image-text retrieval. The image-voice retrieval shows greater value in remote sensing applications than image-text retrieval.…”

Multilanguage Transformer for Improved Text to Remote Sensing Image Retrieval

“…Motivated by this, RS image retrieval has been explored using queries of different modalities such as images from different sources or sensors [11], [12], [13], [14], sketches [15], [16], speech [17], [18], [19], [20], [21], and text [22], [23], [24], [25], [26], [27]. Among these modalities, textual descriptions represent the most intuitive way of communicating with machines.…”

“…In the literature, only few works have been developed for text-image retrieval [22], [23], [24], [25], [26]. All of these works have been designed to allow English as the primary language of the query.…”