High-quality hyperspectral reconstruction using a spectral prior

Abstract

We present a novel hyperspectral image reconstruction algorithm, which overcomes the long-standing tradeoff between spectral accuracy and spatial resolution in existing compressive imaging approaches. Our method consists of two steps: First, we learn nonlinear spectral representations from real-world hyperspectral datasets; for this, we build a convolutional autoencoder which allows reconstructing its own input through its encoder and decoder networks. Second, we introduce a novel optimization method, which jointly regularizes the fidelity of the learned nonlinear spectral representations and the sparsity of gradients in the spatial domain, by means of our new fidelity prior. Our technique can be applied to any existing compressive imaging architecture, and has been thoroughly tested both in simulation, and by building a prototype hyperspectral imaging system. It outperforms the state-of-the-art methods from each architecture, both in terms of spectral accuracy and spatial resolution, while its computational complexity is reduced by two orders of magnitude with respect to sparse coding techniques. Moreover, we present two additional applications of our method: hyperspectral interpolation and demosaicing. Last, we have created a new high-resolution hyperspectral dataset containing sharper images of more spectral variety than existing ones, available through our project website.

KRISM—Krylov Subspace-based Optical Computing of Hyperspectral Images

“…Evaluation against snapshot techniques. We compare KRISM with varying rank against results from [Lin et al 2014a] and [Choi et al 2017] in terms of compression as well as accuracy. We show zoomed in image patches for each method and spectrum at pixel marked by a cross.…”

“…KRISM enjoys advantages when we look at computational cost for reconstruction. The reconstruction time for Choi et al [2017] is more than 10 minutes 1 even with multiple GPUs, while it runs to several hours for Lin et al [2014a] 2 . In contrast, KRISM requires practically no reconstruction time for recovering the HSI as we directly measure the singular vectors.…”

“…We downsampled the HSI to 256 × 256 × 260 to keep computation with CASSI-type simulations tractable. We also used datasets from Choi et al [2017]; Yasuma et al [2010] and Chakrabarti and Zickler [2011] with 31 spectral bands to compare with learning-based techniques. Finally, we present one example from NASA AVIRIS to compare KRISM against Row/Column CS proposed in [Fazel et al 2008].…”

Deep Joint Design of Color Filter Arrays and Demosaicing

“…An autoencoder is a variation of a neural network that tries to learn a representation of its input in a lower‐dimensional space and then reproduce the original information from such a sparse representation. Introduced in the CNN literature as a data‐driven compression method [KW13], the autoencoder concept has already been used for image denoising [VLL*10], data visualization [vdMH08], superresolution [ZYW*15], and to learn priors used for image reconstruction [CJN*17].…”

DeepToF

“…Recently, CAEs have been successfully used in many vision and imaging tasks (e.g. [Choi et al 2017;Du et al 2016]). A straight-forward CAE, nevertheless, cannot be applied to our particular problem: as the errors introduced by MPI are highly correlated with the reference depth to be recovered, we need such ground-truth reference for training.…”