SnapNet-R: Consistent 3D Multi-view Semantic Labeling for Robotics

Abstract

In this paper we present a new approach for semantic recognition in the context of robotics. When a robot evolves in its environment, it gets 3D information given either by its sensors or by its own motion through 3D reconstruction. Our approach uses (i) 3D-coherent synthesis of scene observations and (ii) mix them in a multi-view framework for 3D labeling. (iii) This is efficient locally (for 2D semantic segmentation) and globally (for 3D structure labeling). This allows to add semantics to the observed scene that goes beyond simple image classification, as shown on challenging datasets such as SUNRGBD or the 3DRMS Reconstruction Challenge. * Computed at low resolution (224x224) as in [23] on the contrary of all other results computed at native resolution. **We also test a High Definition strategy, cropping 224x244 patches at original resolution instead of warping image.

Associatively Segmenting Instances and Semantics in Point Clouds

“…Deep Learning on Point Clouds. To take advantage of the strong representation capability of classic CNNs, a 3D point cloud is first projected into multiview rendering images in [33,31,27,9], on which the well-designed CNNs for 2D images can be applied. But part of contextual information in point cloud is left behind during the projection process.…”

LiDAR Point Cloud Recognition of Overhead Catenary System with Deep Learning

“…Inspired by feature learning approaches based on deep learning used for image recognition, researchers proposed some methods to process point cloud. These methods [21][22][23][24] transform the point clouds to voxel-based representation or 2D images as input to deep neural networks, obtaining better recognition results compared with traditional methods based on handcrafted features. However, these methods generally miss much spatial information when transforming and take a long time to train the model.…”

Constraint-based automated reconstruction of grape bunches from 3D range data for high-throughput phenotyping

“…Based on this, Boulch et al (2018) introduced the SnapNet, that renders images from randomly chosen views and uses the U-Net to classify the images, before applying a backprojection to the 3D pointcloud. Guerry et al (2017) optimize the SnapNet for the application on RGB-D data in robotics.…”