Learning Spatially Varying Pixel Exposures for Motion Deblurring

Abstract

Computationally removing the motion blur introduced by camera shake or object motion in a captured image remains a challenging task in computational photography. Deblurring methods are often limited by the fixed global exposure time of the image capture process. The post-processing algorithm either must deblur a longer exposure that contains relatively little noise or denoise a short exposure that intentionally removes the opportunity for blur at the cost of increased noise. We present a novel approach of leveraging spatially varying pixel exposures for motion deblurring using next-generation focal-plane sensor-processors along with an end-to-end design of these exposures and a machine learning-based motion-deblurring framework. We demonstrate in simulation and a physical prototype that learned spatially varying pixel exposures (L-SVPE) can successfully deblur scenes while recovering high frequency detail. Our work illustrates the promising role that focal-plane sensor-processors can play in the future of computational imaging.

Robust Scene Inference under Noise-Blur Dual Corruptions

“…Multi-Exposure Cameras: We demonstrated our approach of multi exposure captures by utilizing multiple cameras with similar (overlapping) fields-of-view. With cameras that are capable of capturing multiple images with varying exposures simultaneously [3], [4], multiple exposure images could be captured with a single camera, thus making it easier to perform spatio-temporal alignment. Our work can be considered as a preliminary proof-of-concept for an eventual implementation where a single camera can capture multiple exposure images.…”

“…For example, modern cell phone cameras can take multiple snaps with a variety of exposures and fuse them to create an aesthetically pleasing image [1]. Increasingly, machine vision sensors [2] are also starting to perform exposure bracketing to capture high dynamic range (HDR) images for autonomous driver assist systems, while others go further and offer the capability of simultaneously capturing different exposure images via a spatially varying exposure sensor for HDR imaging [3] and motion-deblurring [4]. These ongoing developments in camera technology, coupled with the proposed computational techniques can lead to the next generation of computer vision systems which will perform reliably even in non-ideal realworld scenarios (e.g, imagine an autonomous car driving on a dark night attempting to detect pedestrians) where it is extremely challenging for conventional algorithms to extract meaningful information reliably.…”

Video frame interpolation for high dynamic range sequences captured with dual-exposure sensors

“…Dual-exposure sensors [CVM(accessed on Sept. 17, 2021)] offer two motion samples per frame, which, as we show in this work, is greatly beneficial for VFI, albeit requires long-exposure deblurring in the HDR video reconstruction [Heide et al(2014), Cogalan et al(2022), Nguyen et al(2022), Cho et al(2014)]. The scope of these methods is mainly limited to HDR video reconstruction, and they do not aim for the VFI task.…”

“…Programmable sensors with spatially varying exposures [Sony(1999), Carey et al(2013)] greatly expand the dynamic range of contrast in captured scenes [Hajsharif et al(2014), Go et al(2019), Choi et al(2017), Heide et al(2014), Cogalan et al(2022), Nguyen et al(2022), Cho et al(2014)]. In this work, we explore such sensor capabilities toward improving the motion estimation accuracy in VFI.…”