Accurate Spectral Super-Resolution from Single RGB Image Using Multi-scale CNN

Abstract

Different from traditional hyperspectral super-resolution approaches that focus on improving the spatial resolution, spectral superresolution aims at producing a high-resolution hyperspectral image from the RGB observation with super-resolution in spectral domain. However, it is challenging to accurately reconstruct a high-dimensional continuous spectrum from three discrete intensity values at each pixel, since too much information is lost during the procedure where the latent hyperspectral image is downsampled (e.g., with ×10 scaling factor) in spectral domain to produce an RGB observation. To address this problem, we present a multi-scale deep convolutional neural network (CNN) to explicitly map the input RGB image into a hyperspectral image. Through symmetrically downsampling and upsampling the intermediate feature maps in a cascading paradigm, the local and non-local image information can be jointly encoded for spectral representation, ultimately improving the spectral reconstruction accuracy. Extensive experiments on a large hyperspectral dataset demonstrate the effectiveness of the proposed method.

Improving Spatial Resolution of Multispectral Rock Outcrop Images Using RGB Data and Artificial Neural Networks

“…Prior works, in this regard, applied distinct techniques to increase the spatial resolution, varying from sparse data recovery, non-supervised clustering to deep convolution neural networks. To narrow down our scope we considered works that used RGB images as guides to obtain multispectral data like in [ 24 , 32 , 33 , 35 , 50 , 63 ].…”

“…Following this protocol the indexes or coefficients of evaluation takes two images (or set of individual spectral bands), the first is the image with the original resolution and the second is the higher resolution generated image , but resized to the same resolution as the original one, where each multi-spectral element (pixel) in the image is described by i. There are a number of quality indexes to evaluate improved spectral products, that are also used in works that employ data fusion pansharpening or knowledge-based spatial up-sampling, as the RMSE [ 16 , 22 , 49 , 50 ], ERGAS [ 19 , 20 , 49 ], SAM [ 19 , 25 , 31 , 32 , 51 ], UQI [ 14 , 19 , 20 ], and Q4 [ 16 , 19 , 20 ].…”

“…The advances in computational power and a great interest to resolve the high-resolution problems with Super-Resolution (SR) networks based on Convolutional Neural Networks (CNN) have arisen from the work of [ 30 ], which inspired works to improve the spatial resolution of spectral images [ 19 ] and image resolution in general [ 23 , 31 , 32 , 33 , 34 , 35 ], also influenced by the NTIRE [ 36 ] and PIRM2018 [ 37 ] contests. Most of them rely on benchmark datasets or specific datasets available only for the study.…”

Brightness Invariant Deep Spectral Super-Resolution

“…The recent NTIRE2020 challenge on spectral reconstruction from RGB [ 11 ] as well as its predecessor in 2018 [ 12 ] provide a concise overview over the state-of-the-art methods. In summary, the best performing methods in terms of spectral reconstruction quality exclusively consist of complex convolutional neural networks [ 13 , 14 , 15 , 16 , 17 , 18 , 19 ]. A major limitation of deep learning based approaches for SSR is their susceptibility to a varying brightness, i.e., signal scale.…”

Towards Spectral Estimation from a Single RGB Image in the Wild

“…The are works [26,2,4,32,1,10] which employ example-based spectral reconstruction on RGB images. Most common approaches utilize sparse dictionary learning and deep learning methods.…”