Generative Adversarial Learning in YUV Color Space for Thin Cloud Removal on Satellite Imagery

Abstract

Clouds are one of the most serious disturbances when using satellite imagery for ground observations. The semi-translucent nature of thin clouds provides the possibility of 2D ground scene reconstruction based on a single satellite image. In this paper, we propose an effective framework for thin cloud removal involving two aspects: a network architecture and a training strategy. For the network architecture, a Wasserstein generative adversarial network (WGAN) in YUV color space called YUV-GAN is proposed. Unlike most existing approaches in RGB color space, our method performs end-to-end thin cloud removal by learning luminance and chroma components independently, which is efficient at reducing the number of unrecoverable bright and dark pixels. To preserve more detailed features, the generator adopts a residual encoding–decoding network without down-sampling and up-sampling layers, which effectively competes with a residual discriminator, encouraging the accuracy of scene identification. For the training strategy, a transfer-learning-based method was applied. Instead of using either simulated or scarce real data to train the deep network, adequate simulated pairs were used to train the YUV-GAN at first. Then, pre-trained convolutional layers were optimized by real pairs to encourage the applicability of the model to real cloudy images. Qualitative and quantitative results on RICE1 and Sentinel-2A datasets confirmed that our YUV-GAN achieved state-of-the-art performance compared with other approaches. Additionally, our method combining the YUV-GAN with a transfer-learning-based training strategy led to better performance in the case of scarce training data.

Optimization Contrast Enhancement and Noise Reduction for Semantic Segmentation of Oil Palm Aerial Imagery

“…The color space used in the experiment uses BGR to YUV conversion because it is more suitable for recovering cloud-free images. After all, the luminance can change independently without affecting the chromatic information [22].…”

An Overview on the Generation and Detection of Synthetic and Manipulated Satellite Images

“…Reference Year Task Data type Description [65] 2018 Datatype transfer / Inter-modality RGB-SAR Generate optical images from SAR input or fused optical-SAR input [66] 2018 Datatype transfer / Inter-modality SAR-RGB Simulate SAR from optical images [67] 2018 Datatype transfer / Inter-modality RGB-SAR Simulate optical images from SAR images [68] 2018 Datatype transfer / Inter-modality RGB-SAR Simulate optical images from SAR images and vice versa [69] 2019 Datatype transfer / Inter-modality RGB-SAR Simulate optical images from SAR images [70] 2019 Datatype transfer / Inter-modality RGB-SAR Simulate optical images from SAR images [71] 2020 Datatype transfer / Inter-modality RGB Generate optical images from historical maps [72] 2020 Datatype transfer / Intra-modality Multispectral Generate NIR images from RGB images [73] 2020 Datatype transfer / Intra-modality Multispectral Generate certain bands using other bands as input [74] 2018 Quality Improvement / Colorization RGB-SAR Generate colorized SAR images from SARoptical fused image [75] 2019 Quality Improvement / Colorization RGB Adapt color distribution of a testing dataset to match that of a classifier training dataset [76] 2017 Qaulity Improvement / Cloud Removal Multispectral Remove clouds from RGB images using NIR band as auxiliary information [77] 2018 Quality Improvement / Cloud Removal RGB Remove clouds from RGB images [78] 2018 Quality Improvement / Cloud Removal Multispectral-SAR Remove thick clouds from multispectral images [79] 2019 Quality Improvement / Cloud Removal RGB Remove clouds from RGB images [80] 2020 Quality Improvement / Cloud Removal RGB-SAR Remove clouds from RGB images [81] 2020 Quality Improvement / Cloud Removal RGB-SAR Remove clouds from RGB images [82] 2020 Quality Improvement / Cloud Removal Multispectral Remove clouds using temporal data of RGB and NIR bands [83] 2021 Quality Improvement / Cloud Removal RGB Remove clouds from RGB images Table 3. List of methods discussed in Section V and their characteristics.…”

“…Finally, in [83], the authors improve the results of [76] in terms of quality of the generated images considering the YUV color space for the input, instead of the RGB, treating luminance and chroma components independently. As a further difference with respect to [76], they resort to a WGAN [54] architecture which is trained in two-steps.…”

A review of Generative Adversarial Networks (GANs) and its applications in a wide variety of disciplines -- From Medical to Remote Sensing

“…• Cloud Removal: Several authors have used GANs for the removal of clouds contamination from remote sensing images( [147], [97], [128], [175]). CLOUD-GAN [147] can translate cloudy images into cloud-free visible range images.…”