Seeing 3D objects in a single 2D image

Abstract

A general framework simultaneously addressing pose estimation, 2D segmentation, object recognition, and 3D reconstruction from a single image is introduced in this paper. The proposed approach partitions 3D space into voxels and estimates the voxel states that maximize a likelihood integrating two components: the object fidelity, that is, the probability that an object occupies the given voxels, here encoded as a 3D shape prior learned from 3D samples of objects in a class; and the image fidelity, meaning the probability that the given voxels would produce the input image when properly projected to the image plane. We derive a loop-less graphical model for this likelihood and propose a computationally efficient optimization algorithm that is guaranteed to produce the global likelihood maximum. Furthermore, we derive a multi-resolution implementation of this algorithm that permits to trade reconstruction and estimation accuracy for computation. The presentation of the proposed framework is complemented with experiments on real data demonstrating the accuracy of the proposed approach.

A Review of Geometry Recovery from a Single Image Focusing on Curved Object Reconstruction

“…This model imposes strong assumptions on the 3D object, but the dimension of the state space is reduced and only valid 3D reconstructions are obtained. In contrast, Rother and Sapiro [46] impose less strong assumptions on the learned model. For each object class a shape prior is learned as the relative occupancy frequency of each voxel in the object.…”

“…This limits the approach to the reconstruction of buildings in urban environments. In contrast, Rother and Sapiro [46] and Chen and Cipolla [4] shape priors are learned from a database of sample objects. Hence, they are not a-priori limited to a specific object class.…”

“…Rother and Sapiro [46] present a general framework for pose estimation, 2D segmentation, object recognition and 3D reconstruction from a single image. The approach is well-suited to reconstruct bounded objects, but not for elaborate scenes.…”

“…Since such strict assumptions strongly limit the applicability of the approach, most approaches revert to computing the most likely solution to the ill-posed reconstruction problem without guaranteeing accuracy. The probability of a solution is usually measured by means of manually defined [14] or learned shape priors [4,46]. We call this a plausible precision.…”

“…Assumptions for each method are given in column five of Table 16.1. Typical assumptions are a calibrated camera [10], a simplified scene composition [10,17], an object database containing samples for learning shape priors [4,46], a specific viewpoint [41,45,52] or given geometric properties such as vanishing lines of reference planes [8].…”