Towards Spectral Estimation from a Single RGB Image in the Wild

Abstract

In contrast to the current literature, we address the problem of estimating the spectrum from a single common trichromatic RGB image obtained under unconstrained settings (e.g. unknown camera parameters, unknown scene radiance, unknown scene contents). For this we use a reference spectrum as provided by a hyperspectral image camera, and propose efficient deep learning solutions for sensitivity function estimation and spectral reconstruction from a single RGB image. We further expand the concept of spectral reconstruction such that to work for RGB images taken in the wild and propose a solution based on a convolutional network conditioned on the estimated sensitivity function. Besides the proposed solutions, we study also generic and sensitivity specialized models and discuss their limitations. We achieve state-of-the-art competitive results on the standard example-based spectral reconstruction benchmarks: ICVL, CAVE, NUS and NTIRE. Moreover, our experiments show that, for the first time, accurate spectral estimation from a single RGB image in the wild is within our reach. 1

Multispectral Image Reconstruction From Color Images Using Enhanced Variational Autoencoder and Generative Adversarial Network

“…Therefore, we use their work as the primary baseline. We also compare our results with Arad et al's and Berk et al's, since Arad et al claimed their results are state-of-the-art [10] and Berk et al claimed theirs is the first successful estimation of the spectral data from a single RGB image captured in unconstrained settings [14].…”

“…Since it is hard to identify nearby wavelength lights' subtle differences with the naked eyes, the five bands are enough to demonstrate the qualitative evaluation of spectral reconstruction. Furthermore, it is a prevalent way to select several bands to show qualitative results [10], [14]. Other-wise, in the quantitative evaluation section, we demonstrate the full RMSE varieties against the whole spectrum (400nm-700nm) reconstruction results in Figure 7.…”

“…The others two need additional networks to estimate or classify the sensitivities. Moreover, they proved that efficiently estimating the sensitivity function and conditioning the spectral reconstruction model are useful for improving reconstruction accuracy [14]. Although their approach ranked seventh in the NTIRE 2018 Spectral Reconstruction Challenge, they claimed their solution to be the most efficient, with the lowest number of layers and shortest runtime.…”

“…Furthermore, the reconstruction experiment is bi-directional, which means that we measure the results of converting both the RGB to the MSI and the MSI to the RGB. In order to highlight the comparison easily, we include our results with Berk et al's [14], Kin et al's [16], and Arad et al's [10] in the Table 3 and Table 4, respectively.…”

“…Furthermore, we reduce the 32 training images' spatial Arad et al [10], and Berk et al [14], for comparison. Among them, Zhiwei et al's HSCNN claimed their results were state of the art.…”