Jointly Learning Band Selection and Filter Array Design for Hyperspectral Imaging

Abstract

A single-shot multispectral camera equipped with an optimized color filter array (CFA) has the potential to deliver a fast and low-cost hyperspectral (HS) imaging system. Previous solutions are largely restricted to designing demosaicing algorithms for fixed CFAs -be it the Bayer color pattern or evenly-spaced spectral multiplexing patterns. Since sampling and reconstruction are tightly-coupled, the ability to search for an optimal solution is severely constrained by using predefined CFAs. In this work, we simultaneously address the problem of spectral band selection, CFA design, image demosaicing, and spectral image recovery in a joint learning framework for single-shot HS imaging. We propose a reinforcement learning (RL) based method for spectral band selection and a novel neural network for CFA generation, image demosaicing, and HS image recovery. The final spectral reconstruction accuracy is used to supervise the training of the main network to maximize the synergies between those tightly-related tasks. The RL method regards the main network as an agent to collect reward. Our final method delivers a simple setup -as simple as an RGB camera -for HS imaging. Experimental results show that our method outperforms competing methods by a large margin.

InSPECtor: an end-to-end design framework for compressive pixelated hyperspectral instruments

“…The broadband encoding stochastic (BEST) camera descibed in [84] and [85] is developed by a neural network that designs both the spectral filters of a hyperspectral camera and the dense neural network for the reconstruction afterwards. Finally, most closely related to this work is the conference paper by Li et al [86]. They describe the use of a reinforcement-learning-based band selection algorithm in combination with a neural network to design the hyperspectral CFA and do the reconstruction afterwards.…”

LiteCCLKNet: A lightweight criss‐cross large kernel convolutional neural network for hyperspectral image classification

“…Hyperspectral image (HSI) provides rich information for ground objects with hundreds of spectral bands [1]. As a crucial technology of HSI analysis, HSI classification aims to assign a label to each pixel [2].…”

“…The former employs manual feature engineering to design extraction rules applicable to spectral and spatial domains. For example, band selection [1] and feature extraction [6] serving in the spectral domain, local binary pattern [7], and morphological attribute profiles [8] operating on the spatial domain, as well as some extended operators [9, 10] for spectral‐spatial feature fusion. The above methods improve the classification accuracy but require hand‐designed rules, which restricts the applicability and efficiency [11].…”