Multi-Modal Self-Supervised Representation Learning for Earth Observation

Jain, Pallavi; Schoen-Phelan, Bianca; Ross, Robert

doi:10.1109/igarss47720.2021.9553741

Cited by 12 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another work by Chen & Bruzzone, concatenates MS and SAR temporal images to obtain pixel-level discrimination embedding with the distillation network for change detection [24]. Meanwhile, work by Jain et al, demonstrates that single channel features can also provide significant advantages in learning invariant representations for satellite data [3]. Similar results are shown by Montanaro et al [25], which uses a combination of contrastive learning and the text task with the single-channel feature learning.…”

Section: B Ssl and Remote Sensingmentioning

confidence: 77%

“…Similar results are shown by Montanaro et al [25], which uses a combination of contrastive learning and the text task with the single-channel feature learning. Though it has been seen that similarity-based networks such as SimSiam do not perform well in the case of remote sensing [3], [25], we believe that distillation networks provide extra advantages for the learning of invariant spectral and spatial representations.…”

Section: B Ssl and Remote Sensingmentioning

confidence: 80%

“…This problem has motivated many RS domain researchers to either obtain a large amount of labelled land cover data and pre-train a supervised model, or to apply self-supervised learning to the RS data. With the challenges associated with labelled data, self-supervised learning has become quite popular in the RS community [3], [4], [9]- [11].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images

Jain¹,

Schoen-Phelan²,

Ross³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Self-Supervised learning (SSL) has become the new state-of-art in several domain classification and segmentation tasks. Of these, one popular category in SSL is distillation networks such as BYOL. This work proposes RSDnet, which applies the distillation network (BYOL) in the remote sensing (RS) domain where data is non-trivially different from natural RGB images. Since Multi-spectral (MS) and synthetic aperture radar (SAR) sensors provide varied spectral and spatial resolution information, we utilised them as an implicit augmentation to learn invariant feature embeddings. In order to learn RS based invariant features with SSL, we trained RSDnet in two ways, i.e., single channel feature learning and three channel feature learning. This work explores the usefulness of single channel feature learning from random MS and SAR bands compared to the common notion of using three or more bands. In our linear evaluation, these single channel features reached a 0.92 F1 score on the EuroSAT classification task and 59.6 mIoU on the DFC segmentation task for certain single bands. We also compared our results with ImageNet weights and showed that the RS based SSL model outperforms the supervised ImageNet based model. We further explored the usefulness of multi-modal data compared to single modality data, and it is shown that utilising MS and SAR data learn better invariant representations than utilising only MS data.

show abstract

Section: B Ssl and Remote Sensingmentioning

confidence: 77%

Section: B Ssl and Remote Sensingmentioning

confidence: 80%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images

Jain¹,

Schoen-Phelan²,

Ross³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…An improved version [7] described Ace-Net and included a discriminator at latent space level. Another representation learning framework has been detailed in [26] for a classification task. They learnt a common representation by extracting modality specific features with encoders and projecting one feature space to the other using an additional network.…”

Section: Related Workmentioning

confidence: 99%

Deep Learning of Radiometrical and Geometrical Sar Distorsions for Image Modality translations

Bralet

Atto

Chanussot

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Multimodal approaches for Earth Observations suffer from both the lack of interpretability of SAR images and the high sensitivity to meteorological conditions of optical images. Translation methods were implemented to solve them for specific tasks and areas. But these implementations lack of generalizability as they do not include samples with challenging characteristics. Firstly, this paper sums up the main problems that a general SAR to optical image translator should overcome. Then, a SAR Distorted Image to optical translator Network (SARDINet) alternating knowledgeable channel-wise spatial convolutions and cross-channel convolutions is implemented. It aims at solving a problem of major concern in remote sensing: translating layover disturbed SAR images into disturbance-free optical ones. SARDINet is trained through a classical and an adversarial framework and compared to cGAN and cycleGAN from the literature. Experimental results prove that adversarial approaches are more qualitative but worsen quantitative results.

show abstract

“…Contrastive Sensor Fusion [30] constructs positive pairs by using co-located images generated by combining different sensor channels. [31,32,33] apply off-the-shelf contrastive learning algorithms to satellite images. [33] utilizes image inpainting and transformation prediction as additional pretext tasks.…”

Section: Introductionmentioning

confidence: 99%

SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery

Cong¹,

Samar²,

Meng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Developing similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information, we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneficial. Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to ↑ 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to ↑ 14%) and semantic segmentation. * Equal contribution. Order determined via coin flip.Preprint. Under review.

show abstract

Multi-Modal Self-Supervised Representation Learning for Earth Observation

Cited by 12 publications

References 7 publications

Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images

Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images

Deep Learning of Radiometrical and Geometrical Sar Distorsions for Image Modality translations

SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery

Contact Info

Product

Resources

About