Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

Abstract

Cross-modal remote sensing text-image retrieval (RSCTIR) has recently become an urgent research hotspot due to its ability of enabling fast and flexible information extraction on remote sensing (RS) images. However, current RSCTIR methods mainly focus on global features of RS images, which leads to the neglect of local features that reflect target relationships and saliency. In this article, we first propose a novel RSCTIR framework based on global and local information (GaLR), and design a multi-level information dynamic fusion (MIDF) module to efficaciously integrate features of different levels. MIDF leverages local information to correct global information, utilizes global information to supplement local information, and uses the dynamic addition of the two to generate prominent visual representation. To alleviate the pressure of the redundant targets on the graph convolution network (GCN) and to improve the model's attention on salient instances during modeling local features, the denoised representation matrix and the enhanced adjacency matrix (DREA) are devised to assist GCN in producing superior local representations. DREA not only filters out redundant features with high similarity, but also obtains more powerful local features by enhancing the features of prominent objects. Finally, to make full use of the information in the similarity matrix during inference, we come up with a plug-and-play multivariate rerank (MR) algorithm. The algorithm utilizes the k nearest neighbors of the retrieval results to perform a reverse search, and improves the performance by combining multiple components of bidirectional retrieval. Extensive experiments on public datasets strongly demonstrate the state-of-the-art performance of GaLR methods on the RSCTIR task. The code of GaLR method, MR algorithm, and corresponding files have been made available at: https://github.com/xiaoyuan1996/GaLR.

Learning to Evaluate Performance of Multimodal Semantic Localization

“…Next, to improve the inference speed of SeLo tasks, Yuan et al [12] designed a lightweight RSCTIR model, and improved the model from the perspective of knowledge distillation and negative sampling. In [13], the authors qualitatively compare the visualization results of the method and others, thereby verifying the effect of the proposed fusion module. Although RS multimodal semantic localization is a recently emerging task, above works only judge the task from a qualitative analysis perspective, which lacks discriminative quantitative metrics and unified baselines.…”

“…Yuan et al [12] proposed a lightweight text-image retrieval model, which realized fast RS cross-modal retrieval, and improved the retrieval performance by using knowledge extraction and contrast learning. Further, Yuan et al [13] added the denoised detection information to the RS image representation, which greatly improved the retrieval accuracy. The single-stage calculation of embedded-based RSCTIR greatly reduces the loss of information transformation and becomes the main cross-modal retrieval method in recent years.…”

“…In summary, the SeLo can be regarded as a weakly supervised retrieval task with higher-level semantics, which has great application potential in object retrieval. Although studies such as [11][12] [13] have explored this field qualitatively, there is still no unified framework and evaluation basis for this task so far. To provide a quantitative evaluation for SeLo, we construct diverse and reasonable indicators as shown in Fig.…”

“…Vanilla exploration has been carried out on the SeLo task, but the current research is still limited to qualitative analysis [13]. In previous work, researchers qualitatively analyzed the recall metrics of the retrieval models and compared SeLo performance only from a visual perspective.…”

Multimodal Fusion Remote Sensing Image–Audio Retrieval

“…Yuan et al [44] constructed a fine-grained and more challenging remote sensing image-text matching dataset (RSITMD) and proposed a new asymmetric multimodal feature matching network (AMFMN) for the multi-scale scarcity and target redundancy problems in RS multimodal retrieval. Later, Yuan et al [45] proposed a framework for remote sensing image-text matching based on global and local information and designed a multi-level information dynamic fusion module to integrate features at different levels effectively.…”