Self-Supervised Learning for Scene Classification in Remote Sensing: Current State of the Art and Perspectives

Abstract

Deep learning methods have become an integral part of computer vision and machine learning research by providing significant improvement performed in many tasks such as classification, regression, and detection. These gains have been also observed in the field of remote sensing for Earth observation where most of the state-of-the-art results are now achieved by deep neural networks. However, one downside of these methods is the need for large amounts of annotated data, requiring lots of labor-intensive and expensive human efforts, in particular for specific domains that require expert knowledge such as medical imaging or remote sensing. In order to limit the requirement on data annotations, several self-supervised representation learning methods have been proposed to learn unsupervised image representations that can consequently serve for downstream tasks such as image classification, object detection or semantic segmentation. As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.

Enhancing land cover maps with optical time series and ambiguous loss function

“…15,16 The state of the art in SSL was compared and evaluated in more detail in several studies. [16][17][18] A detailed analysis 15 of current SSL solutions and their benefits for agriculture showed the advantages and performance improvements based on different use cases such as weed or crop-type classification, plant disease detection or nitrogen status prediction. For example, Kim et al 19 investigated plant disease detection using SSL and CNN's and achieved up to 14.3% higher accuracies with pre-training.…”

“…generative, predictive and contrastive methods. 17,18 Generative approaches 20 reconstruct noisy or incomplete input data under the assumption that the semantic relationships have been learned. Tseng et al 21 presented a transformer-based model that was pre-trained using four masking strategies and an autoencoder.…”

“…Predictive approaches estimate the applied augmentation to the input and assume that the spatial context is learned implicitly during this process. 17 Examples of pretext tasks are the prediction of the relative position of two image patches or the color of a grey-scale image. 17 Constrastive approaches such as SimCLR or MOCO 23 learn to cluster similar input data close to each other in the embedding space while moving different ones further away.…”

“…17 Examples of pretext tasks are the prediction of the relative position of two image patches or the color of a grey-scale image. 17 Constrastive approaches such as SimCLR or MOCO 23 learn to cluster similar input data close to each other in the embedding space while moving different ones further away. This method has been very successful when using Sentinel-1 and Sentinel-2 data as augmentation 24 for land cover classification.…”

Gradient Structure Information-Guided Attention Generative Adversarial Networks for Remote Sensing Image Generation

“…Generative adversarial networks (GANs) [20] have led to technological breakthroughs in many areas of deep learning and have been rapidly applied in many directions in the field of aerospace remote sensing [21,22]. Remote sensing images usually contain feature information with a large amount of texture and structure information, which is complex [23], and existing natural image generation models rarely consider the structure information in the generation process. Therefore, if they are used in remote sensing image generation, they will lead to geometric structure distortion in the generated sample images, and the generated pseudo-sample images are often poorly realistic and insufficiently diverse to be reliably used as the basis for various analyses and applications in remote sensing [24].…”