A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening

Yuan, Qiangqiang; Wei, Yancong; Meng, Xiangchao; Shen, Huanfeng; Zhang, Liangpei

doi:10.1109/jstars.2018.2794888

Cited by 477 publications

(237 citation statements)

References 46 publications

Supporting

Mentioning

237

Contrasting

Order By: Relevance

“…2020, 12, 993 9 of 24 achieve more accurate result with less distortions. The sparse coefficient matrix α z of Z is obtained by combing (12) and (13).…”

Section: Joint Spatial-spectral Enhancement Algorithmmentioning

confidence: 99%

“…2020, 12, 993 2 of 24 remote sensing data have become increasingly available and the corresponding applications have attracted wide interests, there are existing challenges in acquiring images with simultaneously high spatial resolution and high spectral resolution [6]. Therefore, many research focuses on recovering high quality synthetic image from low resolution (LR) inputs, including spatial resolution improvement approaches [7][8][9][10][11][12][13][14][15][16][17][18][19] and spectral resolution enhancement techniques [20-27].1.1. Spatial Resolution Improvement of HSI Over past decades, several spatial improvement methods have been proposed based on the fusion of MSI and PAN images.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Joint Spatial-spectral Resolution Enhancement of Multispectral Images with Spectral Matrix Factorization and Spatial Sparsity Constraints

et al. 2020

View full text Add to dashboard Cite

This paper presents a joint spatial-spectral resolution enhancement technique to improve the resolution of multispectral images in the spatial and spectral domain simultaneously. Reconstructed hyperspectral images (HSIs) from an input multispectral image represent the same scene in higher spatial resolution, with more spectral bands of narrower wavelength width than the input multispectral image. Many existing improvement techniques focus on spatial-or spectral-resolution enhancement, which may cause spectral distortions and spatial inconsistency. The proposed scheme introduces virtual intermediate variables to formulate a spectral observation model and a spatial observation model. The models alternately solve spectral dictionary and abundances to reconstruct desired high-resolution HSIs. An initial spectral dictionary is trained from prior HSIs captured in different landscapes. A spatial dictionary trained from a panchromatic image and its sparse coefficients provide high spatial-resolution information. The sparse coefficients are used as constraints to obtain high spatial-resolution abundances. Experiments performed on simulated datasets from AVIRIS/Landsat 7 and a real Hyperion/ALI dataset demonstrate that the proposed method outperforms the state-of-the-art spatial-and spectral-resolution enhancement methods. The proposed method also worked well for combination of exiting spatial-and spectral-resolution enhancement methods.Remote Sens. 2020, 12, 993 2 of 24 remote sensing data have become increasingly available and the corresponding applications have attracted wide interests, there are existing challenges in acquiring images with simultaneously high spatial resolution and high spectral resolution [6]. Therefore, many research focuses on recovering high quality synthetic image from low resolution (LR) inputs, including spatial resolution improvement approaches [7][8][9][10][11][12][13][14][15][16][17][18][19] and spectral resolution enhancement techniques [20-27].1.1. Spatial Resolution Improvement of HSI Over past decades, several spatial improvement methods have been proposed based on the fusion of MSI and PAN images. Hyperspectral pan-sharpening improves spatial resolution of HSI by fusing LR HSI with a high resolution (HR) PAN image covering a spectral range from the visible to the near infrared ranges [7]. Except for the classical hyperspectral pan-sharpening approaches such as component substitution (CS) methods [8], multi-resolution analysis (MRA) methods [9], and model based optimization methods [10,11], deep learning, particularly convolutional neural network (CNN), has been widely exploited in pan-sharpening tasks. In [12], a residual CNN model is designed to describe the mapping between LR/HR MSI pairs and PAN image. Yuan et al. [13] proposed a multi-scale and multi-depth CNN (MSDCNN) for pan-sharpening, in which multi-scale feature extraction is used to reconstruct the HR MSI.HSI fusion is another popular approach to spatial resolution improvement. A HR HSI is recovered by fusing a LR HSI with a H...

show abstract

“…2020, 12, 993 9 of 24 achieve more accurate result with less distortions. The sparse coefficient matrix α z of Z is obtained by combing (12) and (13).…”

Section: Joint Spatial-spectral Enhancement Algorithmmentioning

confidence: 99%

mentioning

confidence: 99%

Joint Spatial-spectral Resolution Enhancement of Multispectral Images with Spectral Matrix Factorization and Spatial Sparsity Constraints

et al. 2020

View full text Add to dashboard Cite

show abstract

“…UAV images with a resolution of 1 m or less contain objects of various sizes from very small neighborhoods to large regions composed of thousands of pixels. Smaller features, such as the edges of buildings and the texture of vegetation, tend to be extracted by small-scale convolutional filters, and the coarser general structures tend to respond to larger-scale convolutional filters [19]. In addition, hyperspectral UAV images can provide detailed spectral reflectance signatures, which show electromagnetic energy wavelengths.…”

Section: Network For Very High-resolution Hyperspectral Uav Imagesmentioning

confidence: 99%

Transfer Change Rules from Recurrent Fully Convolutional Networks for Hyperspectral Unmanned Aerial Vehicle Images without Ground Truth Data

Song

Kim

2020

Remote Sensing

View full text Add to dashboard Cite

Change detection (CD) networks based on supervised learning have been used in diverse CD tasks. However, such supervised CD networks require a large amount of data and only use information from current images. In addition, it is time consuming to manually acquire the ground truth data for newly obtained images. Here, we proposed a novel method for CD in case of a lack of training data in an area near by another one with the available ground truth data. The proposed method automatically entails generating training data and fine-tuning the CD network. To detect changes in target images without ground truth data, the difference images were generated using spectral similarity measure, and the training data were selected via fuzzy c-means clustering. Recurrent fully convolutional networks with multiscale three-dimensional filters were used to extract objects of various sizes from unmanned aerial vehicle (UAV) images. The CD network was pre-trained on labeled source domain data; then, the network was fine-tuned on target images using generated training data. Two further CD networks were trained with a combined weighted loss function. The training data in the target domain were iteratively updated using he prediction map of the CD network. Experiments on two hyperspectral UAV datasets confirmed that the proposed method is capable of transferring change rules and improving CD results based on training data extracted in an unsupervised way.

show abstract

“…Multi-scale information could be exploited in mapping learning. In [43], Yuan et al proposed a multi-scale and multi-depth CNN (MSDCNN) for pan-sharpening, whereby each layer was constituted by filters with different sizes for the multi-scale features.…”

Section: Background Of Cnn Based Image Super-resolutionmentioning

confidence: 99%

“…The means to fuse HSI with these auxiliary data in the deep learning framework still lacks study. [38] residual CNN; spectral regularizer is used in loss function SDCNN [39] CNN to learn the spectral difference PNN [40] CNN for pan-sharpening MSI pan-sharpening DRPNN [41] residual CNN for pan-sharpening PanNet [42] residual CNN; learn mapping in high-frequency domain MSDCNN [43] two CNN branches with different depths; multi-scale kernels in each convolutional layer…”

Section: Background Of Cnn Based Image Super-resolutionmentioning

confidence: 99%

Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network

2018

View full text Add to dashboard Cite

Abstract:Enhancing the spatial resolution of hyperspectral image (HSI) is of significance for applications. Fusing HSI with a high resolution (HR) multispectral image (MSI) is an important technology for HSI enhancement. Inspired by the success of deep learning in image enhancement, in this paper, we propose a HSI-MSI fusion method by designing a deep convolutional neural network (CNN) with two branches which are devoted to features of HSI and MSI. In order to exploit spectral correlation and fuse the MSI, we extract the features from the spectrum of each pixel in low resolution HSI, and its corresponding spatial neighborhood in MSI, with the two CNN branches. The extracted features are then concatenated and fed to fully connected (FC) layers, where the information of HSI and MSI could be fully fused. The output of the FC layers is the spectrum of the expected HR HSI. In the experiment, we evaluate the proposed method on Airborne Visible Infrared Imaging Spectrometer (AVIRIS), and Environmental Mapping and Analysis Program (EnMAP) data. We also apply it to real Hyperion-Sentinel data fusion. The results on the simulated and the real data demonstrate that the proposed method is competitive with other state-of-the-art fusion methods.

show abstract

A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening

Cited by 477 publications

References 46 publications

Joint Spatial-spectral Resolution Enhancement of Multispectral Images with Spectral Matrix Factorization and Spatial Sparsity Constraints

Joint Spatial-spectral Resolution Enhancement of Multispectral Images with Spectral Matrix Factorization and Spatial Sparsity Constraints

Transfer Change Rules from Recurrent Fully Convolutional Networks for Hyperspectral Unmanned Aerial Vehicle Images without Ground Truth Data

Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network

Contact Info

Product

Resources

About