Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval

Abstract

Remote sensing (RS) cross-modal text-image retrieval has attracted extensive attention for its advantages of flexible input and efficient query. However, traditional methods ignore the characteristics of multiscale and redundant targets in RS image, leading to the degradation of retrieval accuracy. To cope with the problem of multiscale scarcity and target redundancy in RS multimodal retrieval task, we come up with a novel asymmetric multimodal feature matching network (AMFMN). Our model adapts to multiscale feature inputs, favors multisource retrieval methods, and can dynamically filter redundant features. AMFMN employs the multiscale visual self-attention (MVSA) module to extract the salient features of RS image and utilizes visual features to guide the text representation. Furthermore, to alleviate the positive samples ambiguity caused by the strong intraclass similarity in RS image, we propose a triplet loss function with dynamic variable margin based on prior similarity of sample pairs. Finally, unlike the traditional RS image-text dataset with coarse text and higher intraclass similarity, we construct a fine-grained and more challenging Remote sensing Image-Text Match dataset (RSITMD), which supports RS image retrieval through keywords and sentence separately and jointly. Experiments on four RS text-image datasets demonstrate that the proposed model can achieve state-of-the-art performance in cross-modal RS text-image retrieval task.

Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

“…To obtain a more robust embedding, Abdullah et al [8] proposed a deep bidirectional triplet network to learn the joint encoding between multiple modalities and utilized an averaging fusion strategy to fuse the features of multiple text-image pairs. To solve the problem of multi-scale scarcity and target redundancy in RSCITR, Yuan et al [14] proposed an asymmetric multimodal feature matching network (AMFMN) and contributed a fine-grained RS image-text dataset for this task. By exploring the potential correspondence between RS images and text, Cheng et al [9] proposed a semantic alignment module to get a more discriminative feature representation.…”

“…1) Visual Representation: In the traditional method, CNNs are often used to construct the W v so as to embed the images [41]. Although these methods directly map the global information into the high-dimensional space, they ignore the distinction between significant objects and redundant information due to the complexity of RS image [14]. In addition, the relationship between objects in RS images can not be well represented by global features, which may lead to degradation of retrieval performance.…”

“…Super RS image global features needs to pay attention to the salient object information and suppress useless feature expression. To obtain such global features, the multi-level visual self-attention (MVSA) proposed by Yuan et al [14] came into being. For this purpose, we utilize convolution transformation and MVSA mask to extract global features…”

“…The triplet loss aims to make the distance between the anchor and the negative sample as far as possible, and make it as close as possible to the positive sample. Follow [14], [51], we use the triplet loss as our objective function in our work.…”

“…Due to the heterogeneity gap, the construction of a well-represented unimodal embedding matrix in this method has become an urgent problem for researchers. Yuan et al [14] devised a multi-scale visual self-attention module to filter the salient features in redundant RS images and utilized a cross-modal guidance mechanism to get a superior multi-modal representation. To fully excavate the latent correspondence between RS images and text, Cheng et al [9] proposed a semantic alignment module to filter and optimize data features to achieve higher retrieval accuracy.…”