Learned Semantic Multi-Sensor Depth Map Fusion

Abstract

Volumetric depth map fusion based on truncated signed distance functions has become a standard method and is used in many 3D reconstruction pipelines. In this paper, we are generalizing this classic method in multiple ways: 1) Semantics: Semantic information enriches the scene representation and is incorporated into the fusion process. 2) Multi-Sensor: Depth information can originate from different sensors or algorithms with very different noise and outlier statistics which are considered during data fusion. 3) Scene denoising and completion: Sensors can fail to recover depth for certain materials and light conditions, or data is missing due to occlusions. Our method denoises the geometry, closes holes and computes a watertight surface for every semantic class. 4) Learning: We propose a neural network reconstruction method that unifies all these properties within a single powerful framework. Our method learns sensor or algorithm properties jointly with semantic depth fusion and scene completion and can also be used as an expert system, e.g. to unify the strengths of various photometric stereo algorithms. Our approach is the first to unify all these properties. Experimental evaluations on both synthetic and real data sets demonstrate clear improvements.

Monocular camera and laser based semantic mapping system with temporal-spatial data association for indoor mobile robots

“…Although it provides a real-time and loop-closure solution for indoor semantic interactive system, this work focuses on small-scale structured scenarios. Rozumnyi et al [26] proposed a deep fusion method based on machine learning, which integrated semantic 3D reconstruction, scene construction and multi-sensor data into a learning-based framework. This method automatically extracts sensor parameters and scene attributes parameters from training data and representing them in the form of confidence values to achieve semantic mapping, which only requires a small amount of training data to obtain better generalization ability.…”

UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM

“…Yet, these methods study specific sensors and are not inherently equipped with 3D reasoning. Few works consider 3D reconstruction with multiple sensors [57,31,7,76,77,24], but these do not consider the online mapping setting. Conceptually, more closely related to our work is SenFu-Net [58], which is an online mapping method for multisensor depth fusion.…”

Probabilistic Multi-View Fusion of Active Stereo Depth Maps for Robotic Bin-Picking

“…Hence, they do not account for the fact that, when observing from different viewpoints, depth measurements may present different levels of noise and outliers. To tackle this limitation, [13] and [14] proposed end-to-end learningbased approaches for volumetric depth map fusion. These methods are able to handle sensor noise and outliers, but require a large amount of training data and are relatively prone to over-fitting to a particular sensor or dataset.…”