Rolling Shutter Stereo

Abstract

A huge fraction of cameras used nowadays is based on CMOS sensors with a rolling shutter that exposes the image line by line. For dynamic scenes/cameras this introduces undesired effects like stretch, shear and wobble. It has been shown earlier that rotational shake induced rolling shutter effects in hand-held cell phone capture can be compensated based on an estimate of the camera rotation. In contrast, we analyse the case of significant camera motion, e.g. where a bypassing streetlevel capture vehicle uses a rolling shutter camera in a 3D reconstruction framework. The introduced error is depth dependent and cannot be compensated based on camera motion/rotation alone, invalidating also rectification for stereo camera systems. On top, significant lens distortion as often present in wide angle cameras intertwines with rolling shutter effects as it changes the time at which a certain 3D point is seen. We show that naive 3D reconstructions (assuming global shutter) will deliver biased geometry already for very mild assumptions on vehicle speed and resolution. We then develop rolling shutter dense multiview stereo algorithms that solve for time of exposure and depth at the same time, even in the presence of lens distortion and perform an evaluation on ground truth laser scan models as well as on real street-level data.

Rolling-Shutter-Aware Differential SfM and Image Rectification

“…Due to the price advantage of the RS camera, many researchers began to propose 3D computer vision algorithms that aim to mitigate the RS effect over the recent years. Although several works have successfully demonstrated stereo [21], sparse and dense 3D reconstruction [15,23] and absolute pose estimation [1,22,17] using RS images, most of these works completely bypassed the initial relative pose estimation, e.g. by substituting it with GPS/INS readings.…”

R6P - Rolling shutter absolute pose problem

“…Recent works have shown that RS is an important effect that should be considered in image rectification [10,19], structure from motion [1,8,7] and multiple view stereo [20]. Those works have shown that existing methods can perform poorly on RS data or even fail completely and that incorporating some sort of RS camera model can solve these issues.…”

Direct semi-dense SLAM for rolling shutter cameras

“…Saurer et al [20] showed a RS stereo method which improves accuracy compared to a GS based stereo method. Hedborg et al [8] showed a bundle adjustment method on RS videos.…”

“…Saurer et al [20] proposed a RS stereo method which involves specifying a prior motion model and solving a polynomial equation. Although we could use this, the LSD-SLAM framework separates pose calculation and depth computations (this separation was first very successfully pioneered in PTAM [12]), and this means a simpler solution is available to us, namely finding the generalised epiploar curve as a function of the pose and searching in a manner analogous to any other stereo disparity search.…”