Reconstructing vehicles from a single image: Shape priors for road scene understanding

Abstract

We present an approach for reconstructing vehicles from a single (RGB) image, in the context of autonomous driving. Though the problem appears to be ill-posed, we demonstrate that prior knowledge about how 3D shapes of vehicles project to an image can be used to reason about the reverse process, i.e., how shapes (back-)project from 2D to 3D. We encode this knowledge in shape priors, which are learnt over a small keypoint-annotated dataset. We then formulate a shapeaware adjustment problem that uses the learnt shape priors to recover the 3D pose and shape of a query object from an image. For shape representation and inference, we leverage recent successes of Convolutional Neural Networks (CNNs) for the task of object and keypoint localization, and train a novel cascaded fully-convolutional architecture to localize vehicle keypoints in images. The shape-aware adjustment then robustly recovers shape (3D locations of the detected keypoints) while simultaneously filling in occluded keypoints. To tackle estimation errors incurred due to erroneously detected keypoints, we use an Iteratively Re-weighted Least Squares (IRLS) scheme for robust optimization, and as a by-product characterize noise models for each predicted keypoint. We evaluate our approach on autonomous driving benchmarks, and present superior results to existing monocular, as well as stereo approaches.

Stereo R-CNN Based 3D Object Detection for Autonomous Driving

“…[21] proposes to estimate 3D box using the geometry relations between 2D box edges and 3D box corners. [30,1,22] explicitly utilize sparse information by predicting series of keypoints of regular-shape vehicles. The 3D object pose can be constrained by wireframe template fitting.…”

Disentangling Monocular 3D Object Detection

“…Deep-MANTA [3] uses 3D CAD models and annotated 3D parts in a coarseto-fine localization process. The work in [26] encodes shape priors using keypoints for recovering the 3D pose and shape of a query object. In Mono3D++ [11], the 3D shape and pose for cars is provided by using a morphable wireframe, and it optimizes projection consistency between generated 3D hypotheses and corresponding, 2D pseudomeasurements.…”

The Earth Ain't Flat: Monocular Reconstruction of Vehicles on Steep and Graded Roads from a Moving Camera

“…We also demonstrate that our method is independent of the road plane profile on which vehicles are to be reconstructed. In other words, unlike others (such as [1], [3], [13]) we do not make any assumptions that the ego car and the car to be reconstructed are on the same road plane.…”

“…From top to bottom -(i) Illustrating how co-planarity assumption results in incorrect initialization in existing approaches (ii) Relying only on minimizing the reprojection error, leaves the optimizer free to rigidly transform the mean car (iii) Joint optimization constrains the car to be on ground while minimizing the reprojection error, resulting in more accurate reconstruction and localization (nc and ng are car base and road plane normals respectively) (iv) Failure of co-planarity assumption for steep roads on SYNTHIA-SF [8]. Notice the incorrect initialization of the car on slopes via method of [1] shown in red. Our method is not bound by this co-planarity assumption and initializes the vehicle correctly, shown in black.…”

“…We now formulate a set of cost functions that relax the coplanarity assumptions in [1], [2] and estimate the vehicle's pose and shape as well as the equation of the ground plane patch beneath it.…”