Decoupled Iterative Deep Sensor Fusion for 3D Semantic Segmentation

Abstract

One of the key tasks for autonomous vehicles or robots is a robust perception of their 3D environment, which is why autonomous vehicles or robots are equipped with a wide range of different sensors. Building upon a robust sensor setup, understanding and interpreting their 3D environment is the next important step. Semantic segmentation of 3D sensor data, e.g. point clouds, provides valuable information for this task and is often seen as key enabler for 3D scene understanding. This work presents an iterative deep fusion architecture for semantic segmentation of 3D point clouds, which builds upon a range image representation of the point clouds and additionally exploits camera features to increase accuracy and robustness. In contrast to other approaches, which fuse lidar and camera features once, the proposed fusion strategy iteratively combines and refines lidar and camera features at different scales inside the network architecture. Additionally, the proposed approach can deal with camera failure as well as jointly predict lidar and camera segmentation. We demonstrate the benefits of the presented iterative deep fusion approach on two challenging datasets, outperforming all range image-based lidar and fusion approaches. An in-depth evaluation underlines the effectiveness of the proposed fusion strategy and the potential of camera features for 3D semantic segmentation.

Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation

“…The combination of camera and lidar features is mainly tackled for 3D object detection. The dense fusion of features required for dense predictions like semantic segmentation is only investigated by a few works [18]- [21]. In [18], a dense and region of interest based fusion is applied for multiple tasks, including 3D object detection.…”

“…Afterwards, the concatenated feature maps are fed into LaserNet [22]. Fusion3DSeg [21] applies an iterative fusion strategy for camera and lidar features. Within Fusion3DSeg, camera and range view features are fused following an iterative deep aggregation strategy for an iterative multiscale feature fusion.…”

“…LaserNet++ [22] on the other side only fuses once and does not apply multi-scale fusion. Fusion3DSeg [21] is the most related work and applies an iterative fusion strategy, which considerably differs from our parallel bottom-up and top-down pyramid strategy.…”

“…The lidar backbone computes features for the input point clouds, which are represented in range view, based on the spherical projection of [21], [23]. Its architecture is motivated by EfficientPS [8] and adapted to the range view.…”

“…Additionally, the projection must be applicable at different feature map scales. To solve this task, we apply the scalable projection from Fusion3DSeg [21], [26]. The general idea is to create a mapping from camera image to range view coordinates based on the 3D points of the point cloud.…”