PGMAN: An Unsupervised Generative Multiadversarial Network for Pansharpening

Abstract

Pan-sharpening aims at fusing a low-resolution (LR) multi-spectral (MS) image and a high-resolution (HR) panchromatic (PAN) image acquired by a satellite to generate an HR MS image. Many deep learning based methods have been developed in the past few years. However, since there are no intended HR MS images as references for learning, almost all of the existing methods down-sample the MS and PAN images and regard the original MS images as targets to form a supervised setting for training. These methods may perform well on the down-scaled images, however, they generalize poorly to the full-resolution images. To conquer this problem, we design an unsupervised framework that is able to learn directly from the full-resolution images without any preprocessing. The model is built based on a novel generative multi-adversarial network. We use a twostream generator to extract the modality-specific features from the PAN and MS images, respectively, and develop a dualdiscriminator to preserve the spectral and spatial information of the inputs when performing fusion. Furthermore, a novel loss function is introduced to facilitate training under the unsupervised setting. Experiments and comparisons with other stateof-the-art methods on GaoFen-2, QuickBird and WorldView-3 images demonstrate that the proposed method can obtain much better fusion results on the full-resolution images. Code is available 1 .

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

“…For example, Ma et al (2020) utilize a discriminator to preserve the spatial information using a gradient regularization between the input Pan and spectrally degraded version of the output from generator. The effective loss functions include gradient loss (Seo et al, 2020), perceptual loss (Zhou et al, 2020), and non-reference loss (Zhou et al, 2021a;Luo et al, 2020).…”

LDP-Net: An Unsupervised Pansharpening Network Based on Learnable Degradation Processes

“…Ma et al [20] achieved unsupervised pansharpening using one generator and two discriminators that were designed to distinguish the spatial and spectral characteristics between generated and real images, respectively. Then, Zhou et al [36] combined a generative multi-adversarial network and nonreference loss function to improve the performance of unsupervised pansharpening. Motivated by some priors about downsampling and blurring, several methods have been developed for unsupervised pansharpening.…”

“…3) Comparison Methods: In our experiments, we compared the proposed LDP-Net with several state-of-the-art methods, including PCA [4], IHS [5], Brovey [48], GS [49], BSBD [11], additive wavelet luminance proportional (AWLP) [50], PNN [14], DiCNN [32], PanNet [15], DMDNet [51], FusionNet [33], PGMAN [36] and Pan-GAN [20]. The first six methods belong to traditional method.…”

Remote Sensing Image Fusion Based on Morphological Convolutional Neural Networks with Information Entropy for Optimal Scale

“…Due to the limitations of satellite technology, most remote sensing images can only be panchromatic (PAN) images and low-resolution multispectral (LRMS) images of the same area. The goal of remote sensing image fusion is to fuse the spectral information of LRMS images and the spatial information of PAN images to generate a remote sensing image with both high spatial resolution and high spectral resolution [ 1 ]. Classical component substitution (CS) [ 2 ] methods are the most widely used, but they often result in spectral distortion.…”