Optimized Multi-Spectral Filter Array Based Imaging of Natural Scenes

Abstract

Multi-spectral imaging using a camera with more than three channels is an efficient method to acquire and reconstruct spectral data and is used extensively in tasks like object recognition, relighted rendering, and color constancy. Recently developed methods are used to only guide content-dependent filter selection where the set of spectral reflectances to be recovered are known a priori. We present the first content-independent spectral imaging pipeline that allows optimal selection of multiple channels. We also present algorithms for optimal placement of the channels in the color filter array yielding an efficient demosaicing order resulting in accurate spectral recovery of natural reflectance functions. These reflectance functions have the property that their power spectrum statistically exhibits a power-law behavior. Using this property, we propose power-law based error descriptors that are minimized to optimize the imaging pipeline. We extensively verify our models and optimizations using large sets of commercially available wide-band filters to demonstrate the greater accuracy and efficiency of our multi-spectral imaging pipeline over existing methods.

Optimized Multi-Spectral Filter Arrays for Spectral Reconstruction

“…However, all of these continuous optimizations are not suitable for the discrete combination problem involved filter selection and channel arrangement. In contrast, other overall design methods [3,43] applied heuristic global search algorithm to find a near-optimal MSFA. Motivated by these overall design methods, we model the total errors in the pipeline of imaging and reconstruction by considering the spectral sensitivities of channels, the spatial arrangement of channels, and the statistical prior of spectral images as a whole.…”

“…Sparse recovery for a particular response y can be treated as a two-steps process: first decide the support set for the response; and then optimize a least square problem using the selected support set. Therefore, with a given support set Λ, the coefficients bold-italicαfalse^ can be optimized as:trueα^=(bold-sans-serifΦDΛ)+boldy According to Equation (5), the average spectral reconstruction MSE can be defined as the integral of the estimated error in least square optimization [3], with respect to support set. The MSE is modeled as:MSE≜∫sans-serifΛp(Λ)0.166667emE{||boldDbold-sans-serifΛbold-italicαfalse^−s||22}=∫sans-serifΛp(Λ)0.166667emE{||boldDbold-sans-serifΛ(trueα^−bold-italicα)||22}=∫sans-serifΛp(Λ)0.166667em…”

“…However, most of the existing MSFAs are usually designed in an ad-hoc manner, e.g., the spectral sensitivity function of channels are designed as a series of band-pass functions with a close full width at half maximum (FWHM) and the same intervals. In fact, to design an optimal MSFA, the influence of spectral sensitivity functions of channels, channel spatial distribution, and reconstruction method are mutually correlated [3]. Previous works [4,5,6,7] optimize either (a) the spectral sensitivity functions of channels or (b) the arrangement of channels on periodic mosaic pattern.…”

Spectral Super-Resolution with Optimized Bands

“…A classical approach utilized simulated annealing to optimize the centers and the full-width-half-maximums (FWHMs) of Gaussian MS bands such that those bands produced best reconstruction [25]. Some recent studies have investigated the feasibility of finding optimum MS bands for spectral super-resolution using discrete optimization, where the goal in this approach is to find a subset from a large collection of MS bands which produces the best reconstruction accuracy [26][27][28][29]. Unfortunately, these approaches are not scalable when the number of extracted MS bands becomes much larger than 3 and the spectral range covered is not just visible range, as used in these studies.…”

Plasmonic Color Filter Array with High Color Purity for CMOS Image Sensors

“…These imaging systems in which a filter is externally attached to the image sensor can capture light with a high sensitivity and color purity because of the high utilization efficiency of light but are bulky, occupying a large volume. For downsizing, multispectral filter arrays with different colored filters at each pixel have been studied for multispectral imaging [6,7,8]. Multiband images can be reconstructed using the demasking method by acquiring one channel at each pixel as well as the Bayer color filter array in a conventional RGB camera [9].…”