DOVE: Learning Deformable 3D Objects by Watching Videos

Abstract

Learning deformable 3D objects from 2D images is an extremely ill-posed problem. Existing methods rely on explicit supervision to establish multi-view correspondences, such as template shape models and keypoint annotations, which restricts their applicability on objects "in the wild". In this paper, we propose to use monocular videos, which naturally provide correspondences across time, allowing us to learn 3D shapes of deformable object categories without explicit keypoints or template shapes. Specifically, we present DOVE, which learns to predict 3D canonical shape, deformation, viewpoint and texture from a single 2D image of a bird, given a bird video collection as well as automatically obtained silhouettes and optical flows as training data. Our method reconstructs temporally consistent 3D shape and deformation, which allows us to animate and re-render the bird from arbitrary viewpoints from a single image. IntroductionRecently, with the adoption of machine learning, reconstructing 3D shapes from 2D images has advanced considerably. While this task traditionally requires establishing correspondences between multiple views [13], learning-based approaches have demonstrated the possibility of inferring 3D shapes from a single image, by learning priors for a specific object category [

BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Information

“…Kanazawa et al [7] learn to regress 3D bird shape, given keypoints and silhouettes; birds exhibit rather limited articulation. Recent work obviates the need for 2D keypoints [6,22,26]. Model-based 3D Reconstruction.…”

CASA: Category-agnostic Skeletal Animal Reconstruction

“…Templates are built for various objects including human body [34,1], hand [53], face [29,3], and animals [85,2]. Templatefree methods [19,30,25,64,57,70,23,44,69] study the problem of predicting nonrigid objects directly for a specific category. However, these methods heavily rely on strong category priors such as key-points annotations [19], canonical shapes [10], temporal consistency constraints [30], or canonical surface mapping [23], making them hard to generalize to category-agnostic setting.…”

“…Recent works [74-76, 70, 45] demonstrate promising results. However, they often impose non-realistic assumptions on articulation, such as control point driven deformation [74][75][76]70] or freeform deformation [45,50]. As a result, they fall short of the goal of modeling skeletal characters that can be realistically re-animated in downstream applications.…”

“…1). Unlike existing nonrigid reconstruction works [74,75,70], we exploit a skeleton-based articulation model as well as forward kinematics, ensuring the realism of the resulting skeletal shape ( § 3.1). Specifically, we propose two novel components: a video-to-shape retrieval process and a skeleton-based neural inverse graphics framework.…”

BANMo: Building Animatable 3D Neural Models from Many Casual Videos

“…Category reconstruction from image/video collections. A number of recent methods build deformable 3D models of object categories from images or videos with weak 2D annotations, such as keypoints, object silhouettes, and optical flow, obtained from human annotators or predicted by off-the-shelf models [6,10,14,18,19,52,60]. Such methods often rely on a coarse shape template [16,47], and are not able to recover fine-grained details or large articulations.…”