Hyperspectral Image Super-Resolution with 1D–2D Attentional Convolutional Neural Network

Abstract

Hyperspectral image (HSI) super-resolution (SR) is of great application value and has attracted broad attention. The hyperspectral single image super-resolution (HSISR) task is correspondingly difficult in SR due to the unavailability of auxiliary high resolution images. To tackle this challenging task, different from the existing learning-based HSISR algorithms, in this paper we propose a novel framework, i.e., a 1D-2D attentional convolutional neural network, which employs a separation strategy to extract the spatial-spectral information and then fuse them gradually. More specifically, our network consists of two streams: a spatial one and a spectral one. The spectral one is mainly composed of the 1D convolution to encode a small change in the spectrum, while the 2D convolution, cooperating with the attention mechanism, is used in the spatial pathway to encode spatial information. Furthermore, a novel hierarchical side connection strategy is proposed for effectively fusing spectral and spatial information. Compared with the typical 3D convolutional neural network (CNN), the 1D-2D CNN is easier to train with less parameters. More importantly, our proposed framework can not only present a perfect solution for the HSISR problem, but also explore the potential in hyperspectral pansharpening. The experiments over widely used benchmarks on SISR and hyperspectral pansharpening demonstrate that the proposed method could outperform other state-of-the-art methods, both in visual quality and quantity measurements. a feasible scheme is to increase the pixel spatial size. However, because of fundamental physical limits in practice, it is difficult to improve the sensor capability. Consequently, compared with conventional RGB or multispectral cameras, the obtained hyperspectral image (HSI) is always with a relative low spatial resolution, which limits their practical applications. In recent years, HSI super-resolution (SR) has attracted more and more attention in the remote sensing community.HSI SR is a promising signal post-processing technique aiming at acquiring a high resolution (HR) image from its low resolution (LR) version to overcome the inherent resolution limitations [1]. Generally, we can roughly divide this technique into two categories, according to the availability of an HR auxiliary image, e.g., the single HSI super-resolution or pan sharpening methods with the HR panchromatic image. For example, the popular single HSI super-resolution methods are bilinear [2] and bicubic interpolation [3] based on interpolation, and [4,5] based on regularization. Pansharpening methods can be roughly divided into five categories: component substitution (CS) [6][7][8], which may cause spectral distortion; multiresolution analysis (MRA) [9][10][11][12], which can keep spectral consistency at the cost of much computation and great complexity of parameter setting; bayesian methods [13][14][15] and matrix factorization [16], which can achieve prior spatial and spectral performance at a very high computational cost; and hybrid metho...

Hyperspectral Image Super-Resolution with Self-Supervised Spectral-Spatial Residual Network

“…Recently, many works have designed various deep networks for fusing HR MSIs and LR HSIs, which can be divided into supervised DL-based methods [39] and unsupervised DL-based methods [40].…”

“…To mitigate dependence on the point spread function and spectral response function, Wang et al [24] proposed a blind iterative fusion network to iteratively optimize the observation model. Li et al [39] proposed a twostream network to reconstruct HR HSIs, where one is a 1-D convolutional stream to extract spectral features and the other is a 2-D convolutional stream to extract spatial features. However, in practice, collecting plenty of HR HSIs as supervised information for training is time-consuming and laborious [26,27].…”

Hyperspectral Image Superresolution Using Unidirectional Total Variation With Tucker Decomposition

“…where G pre is the last estimated core tensor. Note that, problem ( 15), ( 17), (19), and ( 20) are all convex. Therefore, we utilize ADMM to solve them.…”

“…Therefore, we utilize ADMM to solve them. Since the solving process of the problem (15), (17), and (19) are similar, to look more concise, we put the solving details of the four problems and each variable updating's computational complexity to Section VII as an appendix.…”

“…During each iteration of ADMM, the heaviest computation steps, shown in (41) Similar to problem (15), problem ( 19) is convex, therefore, we use ADMM to solve it efficiently. We first introduce the splitting variable V 3 = D y U 3 and then rewrite the problem (19) as follows,…”