Multi-Resolution Compressive Spectral Imaging Reconstruction From Single Pixel Measurements

2021

Sensors

Spectral reconstruction (SR) algorithms attempt to recover hyperspectral information from RGB camera responses. Recently, the most common metric for evaluating the performance of SR algorithms is the Mean Relative Absolute Error (MRAE)—an ℓ1 relative error (also known as percentage error). Unsurprisingly, the leading algorithms based on Deep Neural Networks (DNN) are trained and tested using the MRAE metric. In contrast, the much simpler regression-based methods (which actually can work tolerably well) are trained to optimize a generic Root Mean Square Error (RMSE) and then tested in MRAE. Another issue with the regression methods is—because in SR the linear systems are large and ill-posed—that they are necessarily solved using regularization. However, hitherto the regularization has been applied at a spectrum level, whereas in MRAE the errors are measured per wavelength (i.e., per spectral channel) and then averaged. The two aims of this paper are, first, to reformulate the simple regressions so that they minimize a relative error metric in training—we formulate both ℓ2 and ℓ1 relative error variants where the latter is MRAE—and, second, we adopt a per-channel regularization strategy. Together, our modifications to how the regressions are formulated and solved leads to up to a 14% increment in mean performance and up to 17% in worst-case performance (measured with MRAE). Importantly, our best result narrows the gap between the regression approaches and the leading DNN model to around 8% in mean accuracy.

Section: Introductionmentioning

confidence: 99%

On the Optimization of Regression-Based Spectral Reconstruction

2021

Sensors

“…But, there is the overhead of decompressing the signal. Examples include multi-spectral color filter array [13], coded aperture [6,17,18], diffractive gratings [28], digital micro-mirror device [34] and most recently random printed mask [47]. Other problems inherent in compressive sensing are the need for specialized optics and the inherent trade-off between the number of sensors and/or light sensitivity and the spatial resolution.…”

Section: Related Workmentioning

confidence: 99%

“…However, these devices are complicated and/or bulky that limits their usefulness. Other designs deploy novel optical components with specialized post-processing algorithms [13,18,6,17,28,34,47]. But, these devices trade off spatial resolution and/or light sensitivity.…”

Section: Introductionmentioning

confidence: 99%

Physically Plausible Spectral Reconstruction from RGB Images

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

2020

Recently Convolutional Neural Networks (CNN) have been used to reconstruct hyperspectral information from RGB images, and this spectral reconstruction problem (SR) can often be solved with good (low) error. However, little attention has been paid on whether these models' behavior can adhere to physics. We show that the leading CNN method introduces unexpected 'colorimetric errors', which means the recovered spectra do not reproduce ground-truth RGBs, and sometimes this discrepancy can be large. The problem is further compounded by exposure change. Indeed, most CNN models over-fit to fixed exposure and we demonstrate that this can result in poor performance when exposure varies. In this paper we show how CNN learning can be extended so that the physical plausibility of SR is enforced. Remarkably, our physically plausible CNN solutions advance both spectral and colorimetric performance of the original network, while the application of data augmentation trades off the network performance for model stability against varying exposure.

“…But, there is the overhead of decompressing the signal. Examples include multi-spectral color filter array [12], coded aperture [6], [16], [17], diffractive gratings [27], digital micro-mirror device [33] and most recently random printed mask [47]. Other problems inherent in compressive sensing are the need for specialized optics and the inherent trade-off between the number of sensors and/or light sensitivity and the spatial resolution.…”

Section: Related Workmentioning

confidence: 99%

Exposure Invariance in Spectral Reconstruction from RGB Images

2019

cic

Recently Convolutional Neural Networks (CNN) have been used to reconstruct hyperspectral information from RGB images. Moreover, this spectral reconstruction problem (SR) can often be solved with good (low) error. However, these methods are not physically plausible: that is when the recovered spectra are reintegrated with the underlying camera sensitivities, the resulting predicted RGB is not the same as the actual RGB, and sometimes this discrepancy can be large. The problem is further compounded by exposure change. Indeed, most learningbased SR models train for a fixed exposure setting and we show that this can result in poor performance when exposure varies.In this paper we show how CNN learning can be extended so that physical plausibility is enforced and the problem resulting from changing exposures is mitigated. Our SR solution improves the state-of-the-art spectral recovery performance under varying exposure conditions while simultaneously ensuring physical plausibility (i.e. the recovered spectra reintegrate to the input RGBs exactly).