Semi-supervised learning for joint SAR and multispectral land cover classification

Abstract

Self-supervised learning techniques are gaining popularity due to their capability of building models that are effective, even when scarce amounts of labeled data are available. In this paper, we present a framework and specific tasks for self-supervised training of multichannel models, such as the fusion of multispectral and synthetic aperture radar images. We show that the proposed self-supervised approach is highly effective at learning features that correlate with the labels for land cover classification. This is enabled by an explicit design of pretraining tasks which promotes bridging the gaps between sensing modalities and exploiting the spectral characteristics of the input. When limited labels are available, using the proposed self-supervised pretraining and supervised finetuning for land cover classification with SAR and multispectral data outperforms conventional approaches such as purely supervised learning, initialization from training on Imagenet and recent selfsupervised approaches for computer vision tasks.

Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images

“…Similar results are shown by Montanaro et al [25], which uses a combination of contrastive learning and the text task with the single-channel feature learning. Though it has been seen that similarity-based networks such as SimSiam do not perform well in the case of remote sensing [3], [25], we believe that distillation networks provide extra advantages for the learning of invariant spectral and spatial representations.…”

Self-Supervised Learning in Remote Sensing: A review

“…Jung et al [192] combined the sampling idea of tile2vec and contrastive architecture of SimCLR, utilizing smoothed representation of three neighbor tiles as a positive sample. Montanaro et al [193] proposed to use SimCLR for representation learning of the encoder and perturbation invariant autoencoder for segmentation training of the decoder to perform land cover classification. Scheibenreif et al [194] tackled land cover classification and segmentation problems using SimCLR with Swin Transformers and by contrasting optical Sentinel 2 and SAR Sentinel 1 patches.…”

Self-Supervised Vision Transformers for Joint SAR-Optical Representation Learning

“…Two popular modalities in remote sensing are synthetic-aperture radar (SAR) and multispectral (optical) images. Although supervised SARoptical fusion has been extensively studied, corresponding self-supervised techniques are currently in their early stage [14,15,16]. Meanwhile, it is also key to adjust those multimodal algorithms to situations where only a single modality is available.…”