BigEarthNet-MM: A Large-Scale, Multimodal, Multilabel Benchmark Archive for Remote Sensing Image Classification and Retrieval [Software and Data Sets]

Sümbül, Gencer; Wall, Arne; Kreuziger, Tristan; Marcelino, Filipe; Costa, Hugo; Benevides, Pedro; Caetano, Mário; Demir, Begüm; Markl, Volker

doi:10.1109/mgrs.2021.3089174

Cited by 116 publications

(75 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experiments were conducted on the BigEarthNet-MM archive [8] that includes 590,326 multi-modal image pairs. Each pair in BigEarthNet-MM includes one Sentinel-1 (denoted as S1) SAR image and one Sentinel-2 (denoted as S2) multispectral image acquired on the same geographical area.…”

Section: Resultsmentioning

confidence: 99%

A Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing

Sümbül¹,

Müller²,

Demir³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Due to the availability of multi-modal remote sensing (RS) image archives, one of the most important research topics is the development of cross-modal RS image retrieval (CM-RSIR) methods that search semantically similar images across different modalities. Existing CM-RSIR methods require annotated training images (which is time-consuming, costly and not feasible to gather in large-scale applications) and do not concurrently address intra-and inter-modal similarity preservation and inter-modal discrepancy elimination. In this paper, we introduce a novel self-supervised crossmodal image retrieval method that aims to: i) model mutualinformation between different modalities in a self-supervised manner; ii) retain the distributions of modal-specific feature spaces similar; and iii) define most similar images within each modality without requiring any annotated training images. To this end, we propose a novel objective including three loss functions that simultaneously: i) maximize mutual information of different modalities for inter-modal similarity preservation; ii) minimize the angular distance of multi-modal image tuples for the elimination of inter-modal discrepancies; and iii) increase cosine similarity of most similar images within each modality for the characterization of intra-modal similarities. Experimental results show the effectiveness of the proposed method compared to state-of-the-art methods. The code of the proposed method is publicly available at https://git.tu-berlin.de/rsim/SS-CM-RSIR.

show abstract

Section: Resultsmentioning

confidence: 99%

A Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing

Sümbül¹,

Müller²,

Demir³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The experiments were performed on the DLRSD [57] and the BigEarthNet-S2 [58] benchmark archives. The DLRSD archive includes the same images as the UC Merced archive [59] that consists of 2100 aerial images, each of which has the size of 256 × 256 pixels with a spatial resolution of 30 cm.…”

Section: A Dataset Descriptionmentioning

confidence: 99%

“…Each image is annotated with multiple classes provided by the CORINE Land Cover Map (CLC) database of the year 2018. For the experiments, we utilized the 19 class nomenclature presented in [58]. For the tasks that require the availability of land-cover maps, we extracted the CLC land cover map of each image.…”

Section: A Dataset Descriptionmentioning

confidence: 99%

Plasticity-Stability Preserving Multi-Task Learning for Remote Sensing Image Retrieval

Sümbül

Demir

2022

IEEE Trans. Geosci. Remote Sensing

Self Cite

View full text Add to dashboard Cite

Deep learning-based multi-task learning (MTL) methods have recently attracted attention for content-based image retrieval (CBIR) applications in remote sensing (RS). For a given set of tasks (e.g., scene classification, semantic segmentation, and image reconstruction), existing MTL methods employ a joint optimization algorithm on the direct aggregation of task-specific loss functions. Such an approach may provide limited CBIR performance when: 1) tasks compete or even distract each other; 2) one of the tasks dominates the whole learning procedure; or 3) characterization of each task is underperformed compared to single-task learning. This is mainly due to the lack of: 1) plasticity condition (which is associated with sensitivity to new information) or 2) stability condition (which is associated with protection from radical disruptions by new information) of the whole learning procedure. To avoid this issue, as a first time, we propose a novel plasticity-stability preserving MTL (PLASTA-MTL) approach to ensure the plasticity and the stability conditions of the whole learning procedure independently of the number and type of tasks. This is achieved by defining two novel loss functions. The first loss function is the plasticity preserving loss (PPL) function that aims to enforce the global image representation space to be sensitive to new information learned with each task. This is achieved by minimizing the difference of gradient magnitudes for the global representation and task-specific embedding spaces. The second loss function is the stability preserving loss (SPL) function that aims to protect the global representation space radically disrupted by a new task. This is achieved by minimizing the angular distances between the task gradients over global representation space. To effectively employ the proposed loss functions, we also introduce a novel sequential optimization algorithm. Experimental results show the effectiveness of the proposed approach compared to the state-of-the-art MTL methods in the context of CBIR.

show abstract

“…We use the BigEarthNet pre-defined splits as in [21] to train, test and validate our models, based on the 19 land use classes nomenclature. This corresponds to 295,118 (33 GB), 147,559 (17 GB) and 147,559 (17 GB) Sentinel-2 image patches respectively.…”

Section: A Experimental Setupmentioning

confidence: 99%

“…The new classes nomenclature consists of 19 LULC classes [20]. Recently, the dataset was enriched with Synthetic Aperture Radar Sentinel-1 patches [21].…”

Section: Introductionmentioning

confidence: 99%

Efficient deep learning models for land cover image classification

Papoutsis¹,

Bountos²,

Zavras³

et al. 2021

Preprint

View full text Add to dashboard Cite

The availability of the sheer volume of Copernicus Sentinel imagery has created new opportunities for land use land cover (LULC) mapping at large scales using deep learning. Training on such large datasets though is a non-trivial task. In this work we experiment with the BigEarthNet dataset for LULC image classification and benchmark different state-ofthe-art models, including Convolution Neural Networks, Multi-Layer Perceptron, Visual Transformer, EfficientNets and Wide Residual Networks (WRN) architectures. Our aim is to leverage classification accuracy, training time and inference rate. We propose a framework based on EfficientNets for compound scaling of WRNs in terms of network depth, width and input data resolution, for efficiently training and testing different model setups.We design a novel scaled WRN architecture enhanced with an Efficient Channel Attention mechanism. Our proposed lightweight model has an order of magnitude less trainable parameters, achieves 4.5% higher averaged f-score classification accuracy for all 19 LULC classes and is trained two times faster with respect to a ResNet50 state-of-the-art model that we use as a baseline. We provide access to more than 50 trained models, along with our code for distributed training on multiple GPU nodes.

show abstract

BigEarthNet-MM: A Large-Scale, Multimodal, Multilabel Benchmark Archive for Remote Sensing Image Classification and Retrieval [Software and Data Sets]

Cited by 116 publications

References 12 publications

A Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing

A Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing

Plasticity-Stability Preserving Multi-Task Learning for Remote Sensing Image Retrieval

Efficient deep learning models for land cover image classification

Contact Info

Product

Resources

About