Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from RGB

Abstract

Hyperspectral signal reconstruction aims at recovering the original spectral input that produced a certain trichromatic (RGB) response from a capturing device or observer. Given the heavily underconstrained, non-linear nature of the problem, traditional techniques leverage different statistical properties of the spectral signal in order to build informative priors from real world object reflectances for constructing such RGB to spectral signal mapping. However, most of them treat each sample independently, and thus do not benefit from the contextual information that the spatial dimensions can provide. We pose hyperspectral natural image reconstruction as an image to image mapping learning problem, and apply a conditional generative adversarial framework to help capture spatial semantics. This is the first time Convolutional Neural Networks -and, particularly, Generative Adversarial Networks-are used to solve this task. Quantitative evaluation shows a Root Mean Squared Error (RMSE) drop of 44.7% and a Relative RMSE drop of 47.0% on the ICVL natural hyperspectral image dataset.

Deep spectral reflectance and illuminant estimation from self-interreflections

“…Recently, some works aimed at recovering the spectral curves of all the pixels of a single RGB image [21][22][23][24]. The idea of [22] is to model the mapping between camera RGB values and scene reflectance spectra with a radial basis function network.…”

“…At test time, a simple nearest anchor search is run for each RGB triplet in order to reconstruct its spectral curve. Finaly, Alvarez et al proposed a convolutional neural network architecture that learns an end-to-end mapping between pairs of input RGB images and their hyperspectral counterparts [24]. They adopt an adversarial framework-based generative model that takes into account the spatial contextual information present in RGB images for the spectral reconstruction process.…”

Pixel-Aware Deep Function-Mixture Network for Spectral Super-Resolution

“…However, due to the limited expressive capacity of their handcrafted prior models, these methods fail to well generalize to challenging cases. Recently, witnessing the great success of deep convolutional neural networks (DCNNs) in a wide range of tasks (Simonyan and Zisserman 2014; He et al 2016;2017), increasing efforts have been invested to learn a DCNN based mapping function to directly transform the RGB image into an HSI (Alvarez-Gila, Van De Weijer, and Garrote 2017;Arad and Ben-Shahar 2017;Shi et al 2018;Fu et al 2018). These methods essentially involve mapping the RGB context within a size-specific receptive field centered at each pixel to its spectrum in the HSI, as shown in Figure 1.…”

Hyperspectral Image Super-Resolution Using Optimization and DCNN-Based Methods

“…These CNN architectures take only the LR image as input, and the expanding factor of resolution enhancement is theoretically limited to be lower than 8 in both height and width. There are also several works exploring CNNbased method with variant backbone architectures to expand the spectral resolution with only HR-RGB image as input [49,50]. This chapter introduces several research works based on DCNN learning for HS image reconstruction.…”

“…Thus, the reconstructed HS image with acceptable quality usually cannot reach the required spatial resolution for different applications. The spectral resolution enhancement for RGB-to-spectrum reconstruction [49,50] has recently become a hot research line with a single RGB image, which can be lightly collected with a lowprice visual sensor. Although the impressive potential of the RGB-spectrum reconstruction is evaluated, there has still large space for performance improving in real applications.…”