Learning 3D Shape Completion Under Weak Supervision

Abstract

We address the problem of 3D shape completion from sparse and noisy point clouds, a fundamental problem in computer vision and robotics. Recent approaches are either data-driven or learning-based: Datadriven approaches rely on a shape model whose parameters are optimized to fit the observations; Learningbased approaches, in contrast, avoid the expensive optimization step by learning to directly predict complete shapes from incomplete observations in a fullysupervised setting. However, full supervision is often not available in practice. In this work, we propose a weakly-supervised learning-based approach to 3D shape completion which neither requires slow optimization nor direct supervision. While we also learn a shape prior on synthetic data, we amortize, i.e., learn, maximum likelihood fitting using deep neural networks resulting in efficient shape completion without sacrificing accuracy. On synthetic benchmarks based on ShapeNet (Chang et al, 2015) and ModelNet (Wu et al, 2015) as well as on real robotics data from KITTI (Geiger et al, 2012) and Kinect (Yang et al, 2018), we demonstrate that the proposed amortized maximum likelihood approach is able to compete with the fully supervised baseline of Dai et al (2017) and outperforms the data-driven approach of Engelmann et al (2016), while requiring less supervision and being significantly faster.

Leveraging Shape Completion for 3D Siamese Tracking

“…Reconstruction Performances. Table 2 shows shape completion results on the KITTI dataset, using the metrics defined in [50]. Our method (v) outperforms the pure completion one (iii) showing that completion also benefits from the different point of view provided during tracking.…”

“…They regress partial point clouds into full shapes. Alternatively, Stutz et al [50] proposed an occupancy grid shape completion network based on a two-stage training process. Also, Engelmann et al [15] proposed an energy minimization method that aligns shape and pose concurrently in stereo images.…”

Temporally Coherent General Dynamic Scene Reconstruction

“…Research on multiple view dense dynamic reconstruction has primarily focused on indoor scenes with controlled illumination and static backgrounds, extending methods for multiple view reconstruction of static scenes (Seitz et al 2006) to sequences (Tung et al 2009). Deep learning based approaches have been introduced to estimate shape of dynamic objects from minimal camera views in constrained environment (Huang et al 2018;Wu et al 2018) and for rigid objects (Stutz and Geiger 2018). In the last decade, focus has shifted to more challenging outdoor scenes captured with both static and moving cameras.…”

RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion

“…for 2D grids of pixels [21,22,5]. The extensions of CNN in 3D have been shown to work well with 3D voxel grid [38,7,8]. However, the computing cost grows dras- Figure 2: The forward pass of our shape completion network.…”