Hyperspectral Demosaicing of Snapshot Camera Images Using Deep Learning

Abstract

Spectral imaging technologies have rapidly evolved during the past decades. The recent development of single-camera-one-shot techniques for hyperspectral imaging allows multiple spectral bands to be captured simultaneously (3 × 3, 4 × 4 or 5 × 5 mosaic), opening up a wide range of applications. Examples include intraoperative imaging, agricultural field inspection and food quality assessment. To capture images across a wide spectrum range, i.e. to achieve high spectral resolution, the sensor design sacrifices spatial resolution. With increasing mosaic size, this effect becomes increasingly detrimental. Furthermore, demosaicing is challenging. Without incorporating edge, shape, and object information during interpolation, chromatic artifacts are likely to appear in the obtained images. Recent approaches use neural networks for demosaicing, enabling direct information extraction from image data. However, obtaining training data for these approaches poses a challenge as well. This work proposes a parallel neural network based demosaicing procedure trained on a new ground truth dataset captured in a controlled environment by a hyperspectral snapshot camera with a 4 × 4 mosaic pattern. The dataset is a combination of real captured scenes with images from publicly available data adapted to the 4 × 4 mosaic pattern. To obtain real world ground-truth data, we performed multiple camera captures with 1-pixel shifts in order to compose the entire data cube. Experiments show that the proposed network outperforms state-of-art networks.

A Snapshot Multi-Spectral Demosaicing Method for Multi-Spectral Filter Array Images Based on Channel Attention Network

“…To validate the effectiveness of the model, we compare the proposed demosaicing method with three existing traditional demosaicing methods including weighted bilinear interpolation (WB), binary tree-based edge-sensing (BTES), pseudo-panchromatic image difference (PPID), and two deep-learning-based methods, including ResNet-3D [39] and deep convolutional network (DcNet) [40]. Furthermore, three metrics were utilized to represent the quality of the reconstructed multi-spectral image [45,46], which are the peak signal-to-noise ratio (PSNR), the structural similarity (SSIM), and the spectral angle mapper (SAM), respectively.…”

“…However, the images generated by this approach may exhibit false color artifacts in areas of high contrast and brightness. The method proposed by Wisotzky [40] involves using raw sparse multi-spectral images (MSIs) cube as the input of the deep convolutional network (DcNet), maintaining the complete spatial information of the original image. However, performing standard convolution on sparse inputs can lead to some artifacts, making the convergence of the network more difficult.…”