Self-Supervised Vision Transformers for Joint SAR-Optical Representation Learning

Abstract

Self-supervised learning (SSL) has attracted much interest in remote sensing and Earth observation due to its ability to learn task-agnostic representations without human annotation. While most of the existing SSL works in remote sensing utilize ConvNet backbones and focus on a single modality, we explore the potential of vision transformers (ViTs) for joint SAR-optical representation learning. Based on DINO, a state-of-the-art SSL algorithm that distills knowledge from two augmented views of an input image, we combine SAR and optical imagery by concatenating all channels to a unified input. Subsequently, we randomly mask out channels of one modality as a data augmentation strategy. While training, the model gets fed optical-only, SAR-only, and SAR-optical image pairs learning both inner-and intra-modality representations. Experimental results employing the BigEarthNet-MM dataset demonstrate the benefits of both, the ViT backbones and the proposed multimodal SSL algorithm DINO-MM.

Self-Supervised Learning in Remote Sensing: A review

“…Multi-modal/temporal self-supervised learning. As one of the most important characteristics of remote sensing data, multi-modality is a significant aspect to be explored in selfsupervised representation learning [214,220,151]. On the other hand, multi-temporal image analysis is raising more interest because of the increasing frequency of data acquisition and transferring.…”

“…Jain et al [219] perform pre-training by contrasting SAR and optical images using BYOL. Wang et al [220] proposed DINO-MM, a joint SAR-optical representation learning approach with vision transformers. Following the main self-supervised mechanism as DINO, the authors introduced RandomSensorDrop to let the model see all possible combinations of both modalities during training.…”

“…Regarding remote sensing data, a common multi-modality is from optical and SAR images, leading to the important topic SAR-optical data fusion, which has been shown by some recent works that benefits from self-supervised representation learning [214] (see Fig. 22) and [220]. In addition, the fusion of other modalities like audio and social media data is also raising increasing interest [202].…”

Self-Supervised Learning for Scene Classification in Remote Sensing: Current State of the Art and Perspectives

“…Continuing such an approach of joint learning using optical and SAR images, the authors in [96] explore the potential of the non-contrastive method DINO [47] with backbones based on transformer architectures instead of CNNs. As described in Section 2.2.4, DINO is an SSL paradigm that involves a student network trained to extract consistent predictions against a teacher network whose weights are defined as the EMA of the student weights.…”

“…Wang et al 2022 [96] Non-contrastive BigEarthNet-MM Explore DINO paradigm with transformer backbones for optical-SAR joint representation learning in remote sensing. Propose an augmentation which randomly masks either multi-spectral or polarimetric SAR channels.…”