Self-Supervised Material and Texture Representation Learning for Remote Sensing Tasks

Abstract

Self-supervised learning aims to learn image feature representations without the usage of manually annotated labels. It is often used as a precursor step to obtain useful initial network weights which contribute to faster convergence and superior performance of downstream tasks. While selfsupervision allows one to reduce the domain gap between supervised and unsupervised learning without the usage of labels, the self-supervised objective still requires a strong inductive bias to downstream tasks for effective transfer learning. In this work, we present our material and texture based self-supervision method named MATTER (MATerial and TExture Representation Learning), which is inspired by classical material and texture methods. Material and texture can effectively describe any surface, including its tactile properties, color, and specularity. By extension, effective representation of material and texture can describe other semantic classes strongly associated with said material and texture. MATTER leverages multi-temporal, spatially aligned remote sensing imagery over unchanged regions to learn invariance to illumination and viewing angle as a mechanism to achieve consistency of material and texture representation. We show that our self-supervision pretraining method allows for up to 24.22% and 6.33% performance increase in unsupervised and fine-tuned setups, and up to 76% faster convergence on change detection, land cover classification, and semantic segmentation tasks.

Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection

“…Very recently, many contrastive methods have been applied in RS to obtain in-domain pre-trained models that benefit downstream tasks, including land-cover classification [22, 24-28, 60, 61], semantic segmentation [29][30][31][32], and change detection [16][17][18]. Most existing methods apply InfoNCE loss [18, 22, 24-28, 30, 33, 62] or triple loss [17,60] on the constructed positive and negative pairs.…”

“…Some attempts apply SSL directly on a small downstream change detection dataset to extract seasonally invariant features for unsupervised change detection [64][65][66]. Other more related studies that follow the normal SSL pipeline mostly evaluate the pre-trained model on the mediumresolution change detection dataset [16][17][18]. We instead explore pre-trained models suitable for high-resolution RS image change detection.…”

“…Considering the domain gap between natural and RS images may induce sub-optimal representations, a new trend is to pre-train on RS data to better learn the in-domain representations, including supervised pre-training [10,14,15] and selfsupervised learning (SSL) [16][17][18][19]. Contrastive SSL [20,21] could learn useful representations from massive unlabeled data by pulling together representations of semantically similar samples (i.e., positive pairs) and pushing away those of dissimilar samples (i.e., negative pairs).…”

“…Contrastive SSL [20,21] could learn useful representations from massive unlabeled data by pulling together representations of semantically similar samples (i.e., positive pairs) and pushing away those of dissimilar samples (i.e., negative pairs). Very recently, contrastive methods have been introduced in the RS domain [16][17][18][22][23][24][25][26][27][28][29][30][31][32][33] and have shown promising performance for the downstream supervised CD task [16][17][18].…”

Clustering-Based Representation Learning through Output Translation and Its Application to Remote-Sensing Images

“…These pretext tasks include predicting context [25], solving jigsaw puzzles [26,27], image rotation and colorization [28][29][30][31], spatio-temporal consistence [32], and so on. The applications of this method to RSI can be found in [33][34][35], which use domain knowledge and temporal prediction to supervise the training.…”