CloudSEN12, a global dataset for semantic understanding of cloud and cloud shadow in Sentinel-2

Abstract

Accurately characterizing clouds and their shadows is a long-standing problem in the Earth Observation community. Recent works showcase the necessity to improve cloud detection methods for imagery acquired by the Sentinel-2 satellites. However, the lack of consensus and transparency in existing reference datasets hampers the benchmarking of current cloud detection methods. Exploiting the analysis-ready data offered by the Copernicus program, we created CloudSEN12, a new multi-temporal global dataset to foster research in cloud and cloud shadow detection. CloudSEN12 has 49,400 image patches, including (1) Sentinel-2 level-1C and level-2A multi-spectral data, (2) Sentinel-1 synthetic aperture radar data, (3) auxiliary remote sensing products, (4) different hand-crafted annotations to label the presence of thick and thin clouds and cloud shadows, and (5) the results from eight state-of-the-art cloud detection algorithms. At present, CloudSEN12 exceeds all previous efforts in terms of annotation richness, scene variability, geographic distribution, metadata complexity, quality control, and number of samples.

Forecasting localized weather impacts on vegetation as seen from space with meteo-guided video prediction

“…To the best of our knowl-edge, we present the first study considering a climatology baseline and outperforming it with models, which, given the strong seasonality of vegetation dynamics, indicates realworld usefulness of our models in impactful usecases such as humanitarian anticipatory action or carbon monitoring. 1) The baselines reported in table 1 are taken from CloudSEN12 [2]. Sen2Cor [30] is the processing software from ESA used to produce the Scene Classification Layer (SCL) mask, which was also introduced in EarthNet2021 [43].…”

“…The cloudmask in EarthNet2021 is faulty. We train a UNet with Mo-bilenetv2 encoder [48] on the CloudSEN12 dataset [2] to detect clouds and cloud shadows from RGB and Nir bands. Tab.…”

“…works with EarthNet2021) and one with all 13 bands of L2A imagery. We adopted the exact same implementation that was benchmarked in the CloudSEN12 paper [2], with the only difference being that in the paper, L1C imagery was used (which is often not useful in practical use-cases). In detail, this means we trained the UNet with Mobilenetv2 encoder using the Segmentation Models PyTorch Python library 2 .…”

“…We adopted the exact same implementation that was benchmarked in the CloudSEN12 paper [2], with the only difference being that in the paper, L1C imagery was used (which is often not useful in practical use-cases). In detail, this means we trained the UNet with Mobilenetv2 encoder using the Segmentation Models PyTorch Python library 2 . We used a batch size of 32, random horizontal and vertical flipping, random 90 degree rotations, random mirroring, unweighted cross entropy loss, early stopping with a patience of 10 epochs, AdamW optimizer, learning rate of 1e −3 , and a learning rate schedule reducing the learning rate by a factor of 10 if validation loss did not decrease for 4 epochs.…”

“…(1) We expand the EarthNet2021 dataset with a learned cloud mask and a new evaluation scheme to be suitable for vegetation prediction. (2) We present State-of-the-Art models for vegetation prediction, outperforming the top-3 approaches on the EarthNet2021 challenge and multiple other strong baselines both qualitatively and quantitatively. (3) We show how our results can be applied for prognostic carbon monitoring in a real-world use case.…”