Shuffled rolling shutter for snapshot temporal imaging

Abstract

We propose a modification to the rolling shutter mechanism found in CMOS detectors by shuffling the pixels in every scanline. This potential hardware modification improves the sampling of the space-time datacube, allowing the recovery of high-speed videos from a single image using either tensor completion methods or reconstruction algorithms often used for compressive temporal video. We also present a design methodology for optimal sampling schemes and compare them to random shuffling. Simulations, and experimental results obtained by optically emulating the hardware, demonstrate the ability of the shuffled rolling shutter to capture images that allow reconstructing videos, which would otherwise be impossible when using the traditional rolling shutter mechanism.

Multispectral Filter Array Design by Optimal Sphere Packing

“…Thus, solving for the SP problem leads to the optimization of the MSFA, which is analogous to a CA design. In 2022, a hardware modification to the rolling shutter (RS) mechanism found in complementary metal-oxide semiconductor (CMOS) detectors was presented by shuffling the pixels of the scanline, optimizing the position of the sampled pixels within the space-time datacube using SP [9]. The improved sampling scheme and novel reconstruction methods provided an alternative and efficient approach to snapshot temporal imaging.…”

“…From a similar application with spatio-temporal datacubes [9], we realized that we can arrive at optimal solutions to this class of constrained SP design problems by solving a related problem: the 3DN 2 Queens Problem [26]-which is equivalent to finding the position of N 2 Queens within the N 3 slots of a 3D chess board without threatening one to another. Thus, the design of our MSFA-OSP can benefit from the following solution for the placement of the spheres within a kernel B as follows…”

“…2 is all pairwise distance matrix, thereby p k1 and p k2 are the centers of the k 1 th and k 2 th spheres, respectively (see [9]). The positions of the MSFA-OSP are given by…”

“…The neural network in our configuration is an autoencoder with two stages, comprising a set of 3D and 2D convolutional layers. Each convolutional layer is followed by a leaky ReLU activation layer, which has been shown to perform well in many snapshot computational imaging applications [9], [11], [36]. All experiments are run on the PyTorch framework with 2 NVIDIA RTX 3090 GPUs.…”

“…The 3D position of the spectral filters corresponds to the sphere centers, the spatial resolution of the MSFA defines the cubic container size, and the number of filters coincides with the number of packed balls in the cubic container. We denote our approach as Multispectral Filter Array by Optimal Sphere Packing (MSFA-OSP) which has been successfully used for snapshot temporal imaging [9]. Our method maximizes the minimum distance between the spectral filters within the datacube, which can reduce not only artifacts such as color distortions in the demosaicked false color images but also the zipper effect.…”