An Empirical Study of Remote Sensing Pretraining

Wang, Di; Zhang, Jing; Du, B. X.; Xia, Gui-Song; Tao, Dacheng

doi:10.48550/arxiv.2204.02825

Cited by 2 publications

(8 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Considering the domain gap between ImageNet and RS images, a new trend is to pre-train on the remote sensing data to learn the in-domain representations [10, 14-17, 19, 50-53]. Supervised pre-training on RS classification samples [10,14,15,50,51] is still limited by the amount of labeled data, while selfsupervised pre-training could make use of the vast amount of unlabeled RS data [16,17,19].…”

Section: A Handling Label Insufficiency In CDmentioning

confidence: 99%

“…Considering the domain gap between natural and RS images may induce sub-optimal representations, a new trend is to pre-train on RS data to better learn the in-domain representations, including supervised pre-training [10,14,15] and selfsupervised learning (SSL) [16][17][18][19]. Contrastive SSL [20,21] could learn useful representations from massive unlabeled data by pulling together representations of semantically similar samples (i.e., positive pairs) and pushing away those of dissimilar samples (i.e., negative pairs).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection

Chen,

Li,

Chen

et al. 2022

Preprint

View full text Add to dashboard Cite

Training deep learning-based change detection (CD) model heavily depends on labeled data. Contemporary transfer learning-based methods to alleviate the CD label insufficiency mainly upon ImageNet pre-training. A recent trend is using remote sensing (RS) data to obtain in-domain representations via supervised or self-supervised learning (SSL). Here, different from traditional supervised pre-training that learns the mapping from image to label, we leverage semantic supervision in a contrastive manner. There are typically multiple objects of interest (e.g., buildings) distributed in varying locations in RS images. We propose dense semantic-aware pre-training for RS image CD via sampling multiple class-balanced points. Instead of manipulating image-level representations that lack spatial information, we constrain pixel-level cross-view consistency and cross-semantic discrimination to learn spatially-sensitive features, thus benefiting downstream dense CD. Apart from learning illumination invariant features, we fulfill consistent foreground features insensitive to irrelevant background changes via a synthetic view using background swapping. We additionally achieve discriminative representations to distinguish foreground land-covers and other backgrounds. We collect large-scale image-mask pairs freely available in the RS community for pre-training. Extensive experiments on three CD datasets verify the effectiveness of our method. Ours significantly outperforms ImageNet, in-domain supervision, and several SSL methods. Empirical results indicate ours well alleviates data insufficiency in CD. Notably, we achieve competitive results using only 20% training data than random baseline using 100% data. Both quantitative and qualitative results demonstrate the generalization ability of our pre-trained model to downstream images even remaining domain gaps with the pre-training data. Our data and code will make public.

show abstract

Section: A Handling Label Insufficiency In CDmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection

Chen,

Li,

Chen

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, by their experimental results on DIOR [23], they found that knowledge transfer learning also makes the superior detectors in natural scenes obtain competitive results in the RSD without any careful model design. Moreover, Wang, D. et al [34] provided an empirical study of model pre-training in the RSD, which utilized the self-built optical remote sensing dataset M-AID [30] to pre-train models of CNNs [7,10] and ViTs [15,16] and then fine-tuned them on several downstream tasks to avoid the severe domain gap impact of transferring knowledge from the natural scenes to remote sensing scenes. From the experimental results of [34], they found that ViT-based models [15,16] are promising backbones to provide a stronger feature representation to facilitate downstream tasks in the RSD.…”

Section: Knowledge Transfer Learningmentioning

confidence: 99%

“…Therefore, in order to apply these trained data-hungry models for other specific domains, knowledge transfer learning should be considered; specifically, when data-hungry models are applied for remote sensing domain (RSD), they have to utilize a large-scale dataset (e.g., ImageNet [18]) to sufficiently stimulate their potential and then better adapt for various downstream tasks (e.g., scene classification, object detection, and semantic segmentation) in the RSD. Until now, many efforts [19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34] have demonstrated that the consensus solution of supervised pre-training based knowledge transfer learning for model training has basically formed, which needs to pre-train the model on a large-scale dataset with manual annotation and then directly fine-tune pre-trained models on downstream task datasets in the RSD.…”

Section: Introductionmentioning

confidence: 99%

“…Currently, except for knowledge transfer learning via supervised pre-training methods [19,20,[22][23][24][25][26][28][29][30][31][32][33][34]38] in the RSD, refs. [39][40][41] began to explore self-supervised pretraining based knowledge transfer learning, which can avoid massive manually labeling cost and release the potential of unlabeled data for model training in the RSD.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Consecutive Pre-Training: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain

et al. 2022

View full text Add to dashboard Cite

Currently, under supervised learning, a model pre-trained by a large-scale nature scene dataset and then fine-tuned on a few specific task labeling data is the paradigm that has dominated knowledge transfer learning. Unfortunately, due to different categories of imaging data and stiff challenges of data annotation, there is not a large enough and uniform remote sensing dataset to support large-scale pre-training in the remote sensing domain (RSD). Moreover, pre-training models on large-scale nature scene datasets by supervised learning and then directly fine-tuning on diverse downstream tasks seems to be a crude method, which is easily affected by inevitable incorrect labeling, severe domain gaps and task-aware discrepancies. Thus, in this paper, considering the self-supervised pre-training and powerful vision transformer (ViT) architecture, a concise and effective knowledge transfer learning strategy called ConSecutive Pre-Training (CSPT) is proposed based on the idea of not stopping pre-training in natural language processing (NLP), which can gradually bridge the domain gap and transfer large-scale data knowledge to any specific domain (e.g., from nature scene domain to RSD) In addition, the proposed CSPT also can release the huge potential of unlabeled data for task-aware model training. Finally, extensive experiments were carried out on twelve remote sensing datasets involving three types of downstream tasks (e.g., scene classification, object detection and land cover classification) and two types of imaging data (e.g., optical and synthetic aperture radar (SAR)). The results show that by utilizing the proposed CSPT for task-aware model training, almost all downstream tasks in the RSD can outperform the previous knowledge transfer learning strategies based on model pre-training without any expensive manually labeling and even surpass the state-of-the-art (SOTA) performance without any careful network architecture designing.

show abstract

An Empirical Study of Remote Sensing Pretraining

Cited by 2 publications

References 48 publications

Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection

Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection

Consecutive Pre-Training: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain

Contact Info

Product

Resources

About