MantissaCam: Learning Snapshot High-dynamic-range Imaging with Perceptually-based In-pixel Irradiance Encoding

So, Haley M.; Martel, Julien N. P.; Dudek, Piotr; Wetzstein, Gordon

doi:10.48550/arxiv.2112.05221

Cited by 2 publications

(3 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, once the optics are fabricated, the challenge of calibrating them arises. To alleviate the need for calibration, in-pixel sensing strategies can also be designed in an E2E fashion for general capture [38], HDR [19], [20], and compressive imaging [39]. We propose following this route, learning a spatially varying exposure pattern jointly with a network for the task of motion deblurring.…”

Section: Motion Deblurringmentioning

confidence: 99%

“…Future mobile phones may no longer be limited to the fixed global exposures they have today for capturing scenes with dynamic motion or low lighting. High dynamic range (HDR) imaging, most popularly done with some form of exposure bracketing [18], will struggle less with image alignment since programmable sensors can capture information at multiple exposures in a single snapshot [19], [20]. We demonstrate that motion blur is just one of many tasks that highlight the utility of these powerful sensors.…”

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Learning Spatially Varying Pixel Exposures for Motion Deblurring

Nguyen¹,

Martel²,

Wetzstein³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Computationally removing the motion blur introduced by camera shake or object motion in a captured image remains a challenging task in computational photography. Deblurring methods are often limited by the fixed global exposure time of the image capture process. The post-processing algorithm either must deblur a longer exposure that contains relatively little noise or denoise a short exposure that intentionally removes the opportunity for blur at the cost of increased noise. We present a novel approach of leveraging spatially varying pixel exposures for motion deblurring using next-generation focal-plane sensor-processors along with an end-to-end design of these exposures and a machine learning-based motion-deblurring framework. We demonstrate in simulation and a physical prototype that learned spatially varying pixel exposures (L-SVPE) can successfully deblur scenes while recovering high frequency detail. Our work illustrates the promising role that focal-plane sensor-processors can play in the future of computational imaging.

show abstract

Section: Motion Deblurringmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 98%

Learning Spatially Varying Pixel Exposures for Motion Deblurring

Nguyen¹,

Martel²,

Wetzstein³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To mitigate these adverse effects, a method was proposed in which a pixel array was divided into multiple blocks, and the signal integration time of each block was individually controlled [8]. In another method, CE was demonstrated using the pixel-level control of exposure time [9], [10], [11], [12], [13], [14], [15], [16] in addition to the HDR application. In these methods, the readout path and control circuitry should be arranged within the same plane because of unstacked sensors used; thus, the pixel size is relatively large, and a high resolution is difficult to realize.…”

Section: Introductionmentioning

confidence: 99%

A Vision System With 1-inch 17-Mpixel 1000-fps Block-Controlled Coded-Exposure Stacked-CMOS Image Sensor for Computational Imaging and Adaptive Dynamic Range Control

Hirata

Murata

Arii

et al. 2022

IEEE Open J. Circuits Syst.

View full text Add to dashboard Cite

This study introduces a vision system that can acquire images at high speeds and high resolutions. Image sensors are used not only in digital still cameras but also in various applications that require capturing wide luminance differences beyond human perception. For example, fast, high-resolution object recognition, and motion tracking in automatic driving systems are essential, particularly in dark tunnels or the mid-summer sunshine. However, the resolution, frame rate, pixel size, and dynamic range should be traded off to achieve a high performance in capturing moving objects with a high contrast. We have developed a high-speed vision system with a readout operation of 1000 fps, resolution of 4K × 4K, dynamic range of 110 dB, and fine pixels of 2.7 μm. These characteristics were achieved using several technologies such as 1) coded exposure (CE), which divides the image plane into smaller blocks and controls the exposure time of each block individually, 2) arrangement of analog-to-digital converters in parallel for each block, and 3) three-dimensional wafer stacking, which enables high-density integration of circuits and pixels. The proposed system can be applied in CE-based computational imaging in addition to high-dynamic-range applications for handling both the dark and bright areas in a scene.INDEX TERMS Block parallel, coded exposure (CE), high dynamic range (HDR), high-speed imaging, stacked CMOS image sensor (CIS), vision system.

show abstract

MantissaCam: Learning Snapshot High-dynamic-range Imaging with Perceptually-based In-pixel Irradiance Encoding

Cited by 2 publications

References 38 publications

Learning Spatially Varying Pixel Exposures for Motion Deblurring

Learning Spatially Varying Pixel Exposures for Motion Deblurring

A Vision System With 1-inch 17-Mpixel 1000-fps Block-Controlled Coded-Exposure Stacked-CMOS Image Sensor for Computational Imaging and Adaptive Dynamic Range Control

Contact Info

Product

Resources

About