Video stabilization using epipolar geometry

Abstract

We present a new video stabilization technique that uses projective scene reconstruction to treat jittered video sequences. Unlike methods that recover the full three-dimensional geometry of the scene, this model accounts for simple geometric relations between points and epipolar lines. Using this level of scene understanding, we obtain the physical correctness of 3D stabilization methods yet avoid their lack of robustness and computational costs. Our method consists of tracking feature points in the scene and using them to compute fundamental matrices that model stabilized camera motion. We then project the tracked points onto the novel stabilized frames using epipolar point transfer and synthesize new frames using image-based frame warping. Since this model is only valid for static scenes, we develop a time-view reprojection that accounts for non-stationary points in a principled way. This reprojection is based on modeling the dynamics of smooth inertial object motion in three-dimensional space and allows us to avoid the need to interpolate stabilization for moving objects from their static surrounding. Thus, we achieve an adequate stabilization when both the camera and the objects are moving. We demonstrate the abilities of our approach to stabilize hand-held video shots in various scenarios: scenes with no parallax that challenge 3D approaches, scenes containing non-trivial parallax effects, videos with camera zooming and in-camera stabilization, as well as movies with large moving objects.

Spatially and Temporally Optimized Video Stabilization

“…Many of the examples involve large foreground objects occupying almost the whole video frame, making the stabilization rather challenging. While previous methods [3], [4], [8] all fail in such difficult examples, our technique successfully stabilizes the videos with no visual artifacts. To achieve a fair comparison, we also tested the videos chosen from previous methods to demonstrate that our system not only works well in our selected examples but also in theirs.…”

“…We have compared our results with those stabilized by state-of-the-art techniques, including L1-optimization [8], the subspace [3], and the epipolar [4] methods. The 3D stabilization [1] is not included because the latter two approaches [3], [4] can also handle parallax and have been demonstrated to be more robust.…”

“…However, the streaming factorization requires long feature trajectories to cover the entire factorization window, which do not usually appear in videos containing dynamic backgrounds and large foreground motions. Goldstein and Fattal [4] applied the projective reconstruction to account for simple geometric relations between points and epipolar lines, which raises the robustness of scene modeling. They also derived a time-view point reprojection to stabilize trajectories of moving objects so that the defined warping constraints can be spread more uniformly across the frame to capture the scene shape.…”

“…It potentially breaks down when such trajectories are not enough, which usually occurs in videos suffering from serious blurriness. Goldstein and Fattal developed a method that uses a projective scene reconstruction [4] to handle parallax without 3D reconstruction. Their method, however, cannot work well on videos with strong occlusions and non-Lambertain surfaces.…”

“…We demonstrate the effectiveness of our technique by comparing it to state-of-the-art techniques [3], [4], [8] on a variety of videos containing large parallax and strong occlusions. Experiments show that our method can handle parallax and is robust to challenging examples.…”