Joint Spatial-spectral Resolution Enhancement of Multispectral Images with Spectral Matrix Factorization and Spatial Sparsity Constraints

Abstract

This paper presents a joint spatial-spectral resolution enhancement technique to improve the resolution of multispectral images in the spatial and spectral domain simultaneously. Reconstructed hyperspectral images (HSIs) from an input multispectral image represent the same scene in higher spatial resolution, with more spectral bands of narrower wavelength width than the input multispectral image. Many existing improvement techniques focus on spatial-or spectral-resolution enhancement, which may cause spectral distortions and spatial inconsistency. The proposed scheme introduces virtual intermediate variables to formulate a spectral observation model and a spatial observation model. The models alternately solve spectral dictionary and abundances to reconstruct desired high-resolution HSIs. An initial spectral dictionary is trained from prior HSIs captured in different landscapes. A spatial dictionary trained from a panchromatic image and its sparse coefficients provide high spatial-resolution information. The sparse coefficients are used as constraints to obtain high spatial-resolution abundances. Experiments performed on simulated datasets from AVIRIS/Landsat 7 and a real Hyperion/ALI dataset demonstrate that the proposed method outperforms the state-of-the-art spatial-and spectral-resolution enhancement methods. The proposed method also worked well for combination of exiting spatial-and spectral-resolution enhancement methods.Remote Sens. 2020, 12, 993 2 of 24 remote sensing data have become increasingly available and the corresponding applications have attracted wide interests, there are existing challenges in acquiring images with simultaneously high spatial resolution and high spectral resolution [6]. Therefore, many research focuses on recovering high quality synthetic image from low resolution (LR) inputs, including spatial resolution improvement approaches [7][8][9][10][11][12][13][14][15][16][17][18][19] and spectral resolution enhancement techniques [20-27].1.1. Spatial Resolution Improvement of HSI Over past decades, several spatial improvement methods have been proposed based on the fusion of MSI and PAN images. Hyperspectral pan-sharpening improves spatial resolution of HSI by fusing LR HSI with a high resolution (HR) PAN image covering a spectral range from the visible to the near infrared ranges [7]. Except for the classical hyperspectral pan-sharpening approaches such as component substitution (CS) methods [8], multi-resolution analysis (MRA) methods [9], and model based optimization methods [10,11], deep learning, particularly convolutional neural network (CNN), has been widely exploited in pan-sharpening tasks. In [12], a residual CNN model is designed to describe the mapping between LR/HR MSI pairs and PAN image. Yuan et al. [13] proposed a multi-scale and multi-depth CNN (MSDCNN) for pan-sharpening, in which multi-scale feature extraction is used to reconstruct the HR MSI.HSI fusion is another popular approach to spatial resolution improvement. A HR HSI is recovered by fusing a LR HSI with a H...

Hyperspectral Image Super-Resolution in Arbitrary Input-Output Band Settings

“…There are several works [32,52] exploring spatial and spectral SR recently. They find that compared to spatial and spectral SR separately, the end-to-end model can avoid spectral distortion and spatial inconsistency and lead to better overall results.…”

“…They find that compared to spatial and spectral SR separately, the end-to-end model can avoid spectral distortion and spatial inconsistency and lead to better overall results. Yi's method [52] bases on optimization, while Mei's paper [32] focuses on deep-learning models and discusses about the best way to combine spatial and spectral SR stages. As they show us some good results, these methods are fixed to a certain strict setting and cannot be applied on multiple datasets with different settings.…”

Hyperspectral Image Super-Resolution Based on Spatial Group Sparsity Regularization Unmixing

“…Zhang et al [26] exploited the clustering manifold structure in HS-MS image fusion based on the knowledge that the manifold structure is well-preserved in the spatial domain of the MSI. Some Matrix factorization-based methods considering the spectral basis and spatial structures have been proposed; for example, Chen et al [27] proposed a joint spatial-spectral resolution method with spectral matrix factorization and spatial sparsity constraints. The property that real-world HSI are locally low-rank is used to partition the hyperspectral image into patches and helps the optical computing of HSIs [28].…”

Towards Real-Time Hyperspectral Multi-Image Super-Resolution Reconstruction Applied to Histological Samples

“…In a certain sense, this is by itself an act of gathering information about an object or scene without coming into physical contact. There are many specific cases for which HSI presents an interesting approach, such as satellite remote sensing [ 13 ], food quality assessment [ 14 ], and agriculture [ 15 ], dermatology [ 16 ]. In this publication we focus on medical applications, in particular on computational histology [ 17 ].…”