Multilanguage Transformer for Improved Text to Remote Sensing Image Retrieval

Abstract

Cross-modal text-image retrieval in remote sensing (RS) provides a flexible retrieval experience for mining useful information from RS repositories. However, existing methods are designed to accept queries formulated in the English language only, which may restrict accessibility to useful information for non-English speakers. Allowing multilanguage queries can enhance the communication with the retrieval system and broaden access to the RS information. To address this limitation, this article proposes a multilanguage framework based on transformers. Specifically, our framework is composed of two transformer encoders for learning modality-specific representations, the first is a language encoder for generating language representation features from the textual description, while the second is a vision encoder for extracting visual features from the corresponding image. The two encoders are trained jointly on image and text pairs by minimizing a bidirectional contrastive loss. To enable the model to understand queries in multiple languages, we trained it on descriptions from four different languages, namely, English, Arabic, French, and Italian. The experimental results on three benchmark datasets (i.e., RSITMD, RSICD, and UCM) demonstrate that the proposed model improves significantly the retrieval performances in terms of recall compared to the existing state-of-the-art RS retrieval methods.

Prediction of Diffuse Attenuation Coefficient Based on Informer: A Case Study of Hangzhou Bay and Beibu Gulf

“…The Google Machine Translation team first proposes the Transformer model [19]. This model demonstrates strong sequential data modeling capabilities in the field of natural language processing.…”

Enhancing Document Image Retrieval in Education: Leveraging Ensemble-Based Document Image Retrieval Systems for Improved Precision

“…Furthermore, the realm of information retrieval for textual data has been extensively explored, resulting in the development of numerous text-oriented search engines [7]. AlRahhal et al [8] introduced a multilanguage framework centered around text transformers. This framework consists of two transformer encoders designed to learn representations specific to different modalities.…”

“…Our study considers four distinct directions: horizontal, vertical, diagonal, and anti-diagonal. To quantify the run lengths, we use a logarithmic scale for quantization, which is structured as follows: [1], [2], [3][4], [5][6][7][8], [9][10][11][12][13][14][15][16], [17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32], , [65-128], [129-]. In the context of a binary image, run lengths can be computed separately for both the background and foreground.…”

Vision-Language Models for Zero-Shot Classification of Remote Sensing Images

“…One of the key advantages of CLIP is its ability to generalize to new tasks and classes without the need for additional training, making it powerful and flexible enough to perform zero-shot learning. It is worth recalling that these models have been successfully applied to RS tasks related to RS image-text retrieval [3] and visual question answering [4,5].…”