Polarization image demosaicing and RGB image enhancement for a color polarization sparse focal plane array

Abstract

The color division of focal plane (DoFP) polarization sensor structure mostly uses Bayer filter and polarization filter superimposed on each other, which makes the polarization imaging unsatisfactory in terms of photon transmission rate and information fidelity. In order to obtain high-resolution polarization images and high-quality RGB images simultaneously, we simulate a sparse division of focal plane polarization sensor structure, and seek a sweet spot of the simultaneous distribution of the Bayer filter and the polarization filters to obtain both high-resolution polarization images and high-quality RGB images. In addition, From the perspective of sparse polarization sensor imaging, leaving aside the traditional idea of polarization intensity interpolation, we propose a new sparse Stokes vector completion method, in which the network structure avoids the introduction and amplification of noise during polarization information acquisition by mapping the S1 and S2 components directly. The sparsely polarimetric image demosaicing (Sparse-PDM) model is a progressive combined structure of RGB image artifact removal enhancement network and sparsely polarimetric image completion network, which aims to compensate sparsely polarimetric Stokes parameter images with the de-artifacts RGB image as a guide, thus achieving high-quality polarization information and RGB image acquisition. Qualitative and quantitative experimental results on both self-constructed and publicly available datasets prove the superiority of our method over state-of-the-art methods.

Low-Light Sparse Polarization Demosaicing Network (LLSPD-Net): Polarization Image Demosaicing Based on Stokes Vector Completion in Low-Light Environment

“…Our second aim concerns the fact that DoFP sensors have a low photon transmittance due to their own structure, and polarization demodulation often requires a process of “sampling + interpolation + computation”. This results in severe noise and distortion, so we compensate for missing pixels in the Stokes vector map directly through a deep learning network with the help of a sparse sensor structure proposed in the literature [ 25 ]. This method is able to ensure high-quality RGB images while obtaining polarized images with less noise.…”

“…The dataset takes into full consideration the large polarization difference between metal and dielectric materials, etc., and collects indoor and outdoor scene data with different shapes and materials, such as metal, plastic, wood, and fabric. Among them, the sparse polarization images used in this paper are acquired from reference [ 25 ]. In order to more clearly reflect the diversity of the training data in the dataset of this paper, we crop the images of the dataset into 64 × 64 subimages and map these subimages to a point within the unit circle.…”

Passive Polarized Vision for Autonomous Vehicles: A Review

“…With such devices, we obtain 12-channel mosaiced images, the information is then rather sparse for each channel. Efficient demosaicing algorithms are required to prevent color and polarization reconstruction artifacts [ 125 , 144 , 145 ]. Alternate geometries combining CFA and PFA were proposed in order to maximize the signal to noise ratio or minimize the reconstruction artifacts [ 145 , 146 , 147 ].…”

A Super-Resolution Reconstruction Method for Infrared Polarization Images with Sparse Representation of Over-Complete Basis Sets

“…Super-resolution reconstruction techniques can be categorized based on their implementation methods into interpolation methods [12][13][14][15][16], reconstruction methods [17][18][19], and learning methods [20][21][22][23][24][25][26][27][28]. The interpolation methods include bilinear and bicubic interpolation methods [12], which were the initial algorithms utilized for super-resolution tasks.…”

“…An objective evaluation index indicated the superior performance of FORK-NET compared to PDCNN. Consequently, researchers explored alternative network models, such as the Multi-Scale Adaptive Weighted Network (MSAWN) [22], Deep Compressed Sensing (DCS) [23], and the sparsely polarimetric image demosaicing model (Sparse-PDM) [24]. These networks primarily target visible light polarization and necessitate extensive training data, typically sourced from Sony IMX250 series visible light polarization cameras.…”