GSC-MIM: Global semantic integrated self-distilled complementary masked image model for remote sensing images scene classification

Abstract

Masked image modeling (MIM) is a learning method in which the unmasked components of the input are utilized to learn and predict the masked signal, enabling learning from large amounts of unannotated data. However, due to the scale diversity and complexity of features in remote sensing images (RSIs), existing MIMs face two challenges in the RSI scene classification task: (1) If the critical local patches of small-scale objects are randomly masked out, the model will be unable to learn its representation. (2) The reconstruction of MIM relies on the visible local contextual information surrounding the masked regions and overemphasizing this local information will potentially lead the model to disregard the global semantic information of the input RSI. Regarding the above considerations, we proposed a global semantic integrated self-distilled complementary masked image model (GSC-MIM) for RSI scene classification. To prevent information loss, we proposed an information-preserved complementary masking strategy (IPC-Masking), which generates two complementary masked views for the same image to resolve the problem of masking critical areas of small-scale objects. To incorporate global information into the MIM pre-training process, we proposed the global semantic distillation strategy (GSD). Specifically, we introduced an auxiliary network pipeline to extract the global semantic information from the full input RSI and transfer the knowledge to the MIM by self-distillation. The proposed GSC-MIM is validated on three publicly available datasets of AID, NWPU-RESISC45, and UC-Merced Land Use, and the results show that the proposed method's Top-1 accuracy surpasses the baseline approaches in three datasets by up to 4.01, 3.87, and 5.26%, respectively.

Deep Learning Methods for Semantic Segmentation in Remote Sensing with Small Data: A Survey

“…Based on this, Sun et al [103] proposed that RingMo uses the incomplete strategy method to improve the capture of small objects. Similar studies have been conducted [144], which have simultaneously reconstructed complementary masked-visible-region views and incorporated a global semantic distillation strategy (GSD) to ensure that salient areas of small objects are not lost. Muhtar et al [99] constructed a knowledge distillation network that inputs data-enhanced images to the teacher network and masked images to the student network for updated learning, and this method aims to accomplish the SSL task using the MIM.…”