SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery

Abstract

Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Developing similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information, we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneficial. Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to ↑ 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to ↑ 14%) and semantic segmentation. * Equal contribution. Order determined via coin flip.Preprint. Under review.

GFM: Building Geospatial Foundation Models via Continual Pretraining

“…WHU Aerial Vaihingen ResNet50 (ImageNet-1k) [21] 88.5 74.0 SeCo [31] 86.7 68.9 ViT (ImageNet-22k) [15] 81.6 72.6 SatMAE [10] 82.5 70.6 Swin (random) [28] 88.2 67.0 Swin (ImageNet-22k) [28] 90.4 74.7 GFM 90.7 75.3…”

“…[40] employs a colorization objective on Sentinel-2 imagery utilizing the various spectral bands. Most recently, SatMAE [10] explores the use of masked image modeling to train a large ViT model. This work is similar in some respect to ours, as we also train a transformer model with an MIM objective.…”

“…The data consist of 12band Sentinel-2 images with sizes of 120x120, 60x60, and 20x20 pixels for the bands at 10m, 20m, and 60m GSDs, re- spectively. We employ the data split and 19 class evaluation as common in the literature [10,31,32].…”

“…Nonetheless, in the current era of deep * Work done as an intern at Amazon Web Services Figure 1. Our sustainable geospatial foundation model (GFM) achieves strong performance on a broad set of tasks in comparison to other state-of-the-art geospatial pretraining methods (SeCo [31], SatMAE [10]) and ImageNet supervised pretraining baselines.…”

“…With such a vast interest in the application of geospatial and remote sensing data, the computer vision community has been continually investing in designing better algorithms to harness the available data and improve performance on various tasks. Particularly, with the rising popularity of foundation models in vision and natural language, many works have worked on building strong pretrained models specific to the geospatial domain [2,10,31,41]. These methods typically train a network from scratch on a large corpus of remote sensing imagery.…”