Reconstruction of Articulated Objects from a Moving Camera

Abstract

Many scenes that we would like to reconstruct contain articulated objects, and are often captured by only a single, non-fixed camera. Existing techniques for reconstructing articulated objects either require templates, which can be challenging to acquire, or have difficulties with perspective effects and missing data. In this paper, we present a novel reconstruction pipeline that first treats each feature point tracked on the object independently and incrementally imposes constraints. We make use of the idea that the unknown 3D trajectory of a point tracked in 2D should lie on a manifold that is described by the camera rays going through the tracked 2D positions. We compute an initial reconstruction by solving for latent 3D trajectories that maximize temporal smoothness on these manifolds. We then leverage these 3D estimates to automatically segment an object into piecewise rigid parts, and compute a refined shape and motion using sparse bundle adjustment. Finally, we apply kinematic constraints on automatically computed joint positions to enforce connectivity between different rigid parts, which further reduces ambiguous motion and increases reconstruction accuracy. Each step of our pipeline enforces temporal smoothness, and together results in a high quality articulated object reconstruction. We show the usefulness of our approach in both synthetic and real datasets and compare against other non-rigid reconstruction techniques.

Post-Estimation Smoothing: A Simple Baseline for Learning with Side Information

“…Recent work has shown that improvements in human pose estimation [13,21,41] and object detection and classification in videos [28,40,47] can be obtained by encoding temporal consistency as part of a larger predictive pipeline. The intuition is that exploiting continuity of motion over video frames can reduce the noise in per-frame predictions.…”

Reconstructing Articulated Rigged Models from RGB-D Videos

“…Reconstructing articulated objects has attracted a lot of interest during the past decade. Due to the popularity of different image sensors over the years, research focus has gradually shifted from reconstructing 2D skeletons from RGB data [8,9,10,11] to 3D skeletons from RGB [12,13,14,15] or RGB-D data [16,4,17].…”

“…In contrast to [18], it is able to reconstruct complex articulated structures. [15] use ray-space optimization to estimate 3D trajectories from 2D trajectories. The approach, however, assumes that the num-ber of parts is known.…”