Photosequencing of Motion Blur using Short and Long Exposures

Rengarajan, Vijay; Zhao, Su; Zhen, Ruiwen; Glotzbach, John William; Sheikh, H.R.; Sankaranarayanan, Aswin C.

doi:10.1109/cvprw50498.2020.00263

Cited by 14 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Elmalem et al [8] design an optic that encodes blur in its color PSF, and Yosef et al [9] build on this work by additionally using a focus mechanism to recover video frames from a capture with motion blur. Rengarajan et al [42] proposed using short-long-short exposures with recursive blur decomposition to deblur. Notably, all the above approaches have been designed using heuristics and theory, making the search space limited to human intuition.…”

Section: Motion Deblurringmentioning

confidence: 99%

Learning Spatially Varying Pixel Exposures for Motion Deblurring

Nguyen¹,

Martel²,

Wetzstein³

2022

Preprint

View full text Add to dashboard Cite

Computationally removing the motion blur introduced by camera shake or object motion in a captured image remains a challenging task in computational photography. Deblurring methods are often limited by the fixed global exposure time of the image capture process. The post-processing algorithm either must deblur a longer exposure that contains relatively little noise or denoise a short exposure that intentionally removes the opportunity for blur at the cost of increased noise. We present a novel approach of leveraging spatially varying pixel exposures for motion deblurring using next-generation focal-plane sensor-processors along with an end-to-end design of these exposures and a machine learning-based motion-deblurring framework. We demonstrate in simulation and a physical prototype that learned spatially varying pixel exposures (L-SVPE) can successfully deblur scenes while recovering high frequency detail. Our work illustrates the promising role that focal-plane sensor-processors can play in the future of computational imaging.

show abstract

Section: Motion Deblurringmentioning

confidence: 99%

Learning Spatially Varying Pixel Exposures for Motion Deblurring

Nguyen¹,

Martel²,

Wetzstein³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Blur-to-video methods [3,22,23,49,61,88] leverage motion blur and recover the image sequence, constraining the optimization with spatial sharpness and temporal smoothness. To regularize the loss of texture during capture, Yuan et al [86] use a long and short exposure pair for deblurring, while Rengarajan et al [53] exploit the idea to reconstruct high-speed videos. Coded exposure methods replace the box-filter averaging over time with a broadband filter averaging by switching the shutter on and off multiple times with varying on-off durations within a single exposure period.…”

Section: Video Downsamplingmentioning

confidence: 99%

Learning Spatio-Temporal Downsampling for Effective Video Upscaling

Xiang¹,

Tian²,

Rengarajan³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Downsampling is one of the most basic image processing operations. Improper spatio-temporal downsampling applied on videos can cause aliasing issues such as moiré patterns in space and the wagon-wheel effect in time. Consequently, the inverse task of upscaling a low-resolution, low frame-rate video in space and time becomes a challenging ill-posed problem due to information loss and aliasing artifacts. In this paper, we aim to solve the space-time aliasing problem by learning a spatio-temporal downsampler. Towards this goal, we propose a neural network framework that jointly learns spatio-temporal downsampling and upsampling. It enables the downsampler to retain the key patterns of the original video and maximizes the reconstruction performance of the upsampler. To make the downsamping results compatible with popular image and video storage formats, the downsampling results are encoded to uint8 with a differentiable quantization layer. To fully utilize the spacetime correspondences, we propose two novel modules for explicit temporal propagation and space-time feature rearrangement. Experimental results show that our proposed method significantly boosts the space-time reconstruction quality by preserving spatial textures and motion patterns in both downsampling and upscaling. Moreover, our framework enables a variety of applications, including arbitrary video resampling, blurry frame reconstruction, and efficient video storage.

show abstract

“…Computational only [19,35,54] conventional image fixed Multiple exposures [37] short-long-short exposures 3 images dynamic Coded two-bucket [39,41] C2B sensor two coded images fixed Event camera [34] event sensor 50-100 events vector dynamic Proposed method dynamic phase coding coded image dynamic Table 1. Overview of existing solutions for video reconstruction from a motion-blurred scene.…”

Section: Related Workmentioning

confidence: 99%

“…A closely related problem is a reconstruction of a sharp and high frame rate video from motion blurred and low frame rate video, using either processing of conventional camera videos [18,37] or computational imaging methods [15,25,26]. These methods require a video input (which enables to solve the direction ambiguity), and are not applicable for single image input.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Video Reconstruction from a Single Motion Blurred Image using Learned Dynamic Phase Coding

Yosef¹,

Elmalem²,

Giryes³

2021

Preprint

View full text Add to dashboard Cite

Video reconstruction from a single motion-blurred image is a challenging problem, which can enhance existing cameras' capabilities. Recently, several works addressed this task using conventional imaging and deep learning. Yet, such purely-digital methods are inherently limited, due to direction ambiguity and noise sensitivity. Some works proposed to address these limitations using non-conventional image sensors, however, such sensors are extremely rare and expensive. To circumvent these limitations with simpler means, we propose a hybrid optical-digital method for video reconstruction that requires only simple modifications to existing optical systems. We use a learned dynamic phase-coding in the lens aperture during the image acquisition to encode the motion trajectories, which serve as prior information for the video reconstruction process. The proposed computational camera generates a sharp frame burst of the scene at various frame rates from a single coded motion-blurred image, using an image-to-video convolutional neural network. We present advantages and improved performance compared to existing methods, using both simulations and a real-world camera prototype.

show abstract

Photosequencing of Motion Blur using Short and Long Exposures

Cited by 14 publications

References 24 publications

Learning Spatially Varying Pixel Exposures for Motion Deblurring

Learning Spatially Varying Pixel Exposures for Motion Deblurring

Learning Spatio-Temporal Downsampling for Effective Video Upscaling

Video Reconstruction from a Single Motion Blurred Image using Learned Dynamic Phase Coding

Contact Info

Product

Resources

About