GFM: Building Geospatial Foundation Models via Continual Pretraining

Abstract

Geospatial technologies are becoming increasingly essential in our world for a large range of tasks, such as earth monitoring and natural disaster response. To help improve the applicability and performance of deep learning models on these geospatial tasks, various works have pursued the idea of a geospatial foundation model, i.e., training networks from scratch on a large corpus of remote sensing imagery. However, this approach often requires a significant amount of data and training time to achieve suitable performance, especially when employing large state-of-the-art transformer models.In light of these challenges, we investigate a sustainable approach to building geospatial foundation models. In our investigations, we discover two important factors in the process. First, we find that the selection of pretraining data matters, even within the geospatial domain. We therefore gather a concise yet effective dataset for pretraining. Second, we find that available pretrained models on diverse datasets like ImageNet-22k should not be ignored when building geospatial foundation models, as their representations are still surprisingly effective. Rather, by leveraging their representations, we can build strong models for geospatial applications in a sustainable manner. To this end, we formulate a multi-objective continual pretraining approach for training sustainable geospatial foundation models. We experiment on a wide variety of downstream datasets and tasks, achieving strong performance across the board in comparison to ImageNet baselines and state-ofthe-art geospatial pretrained models.

SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing

“…The first line of research aims to learn the representations of remote sensing images through establishing the similarity metrics between multiple images captured at different geolocations or multiple views of the same object/location (Jean et al 2019;Manas et al 2021;Ayush et al 2021). Another line of research attempts to train a vision model with the goal to reconstruct masked patches of the input image (Cong et al 2022;Fuller, Millard, and Green 2022a,b;Reed et al 2022;Mendieta et al 2023;Cha, Seo, and Lee 2023)…”

Deep-Learning-Based Semantic Segmentation of Remote Sensing Images: A Survey

“…The three elements that make up a foundation model include a large-scale dataset, a pretrained model, and the pretraining method. Since 2021, several million-scale publicly available large-scale datasets have been built in remote sensing, including FMoW [249], SeCo [250], Million-AID [251], Levir-KR [252], GeoPile [253], SSL4EO-S12 [254] and SSL4EO-L [255]. In the early days, pretrained model research typically deals with ResNet50 [250], [252], ViT Large [256], [257], Swin Base [253], [258], ViTAE-B [259], and ViTAEv2-S [260], the number of parameters of them was relatively small compared those in computer vision.…”

“…Since 2021, several million-scale publicly available large-scale datasets have been built in remote sensing, including FMoW [249], SeCo [250], Million-AID [251], Levir-KR [252], GeoPile [253], SSL4EO-S12 [254] and SSL4EO-L [255]. In the early days, pretrained model research typically deals with ResNet50 [250], [252], ViT Large [256], [257], Swin Base [253], [258], ViTAE-B [259], and ViTAEv2-S [260], the number of parameters of them was relatively small compared those in computer vision. Recently, with the rapid development of vision foundation models, the size of remote sensing pretrained models has expanded to billon-scale [261].…”

“…Therefore, SSL has become the main method of remote sensing foundation models. Currently, SSL methods in remote sensing mainly include contrastive learning (CS) [250], self-distillation [253], and masked image modeling (MIM) [256], [257], [258], [259], [262], [261]. Contrastive learning constructs positive and negative pairs to learn both invariant and distinguishable visual features [250], [254], [255].…”

MARE: Self-Supervised Multi-Attention REsu-Net for Semantic Segmentation in Remote Sensing

“…More broadly, continual pretraining has been found to improve the performance of self-supervised models on many tasks [6]. As RS data is available as large but unlabeled datasets, this has become a popular technique to specialize large-scale models for remote EO [55]. [56], [57] e.g.…”