LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation

Abstract

Camera and 3D LiDAR sensors have become indispensable devices in modern autonomous driving vehicles, where the camera provides the fine-grained texture, color information in 2D space and LiDAR captures more precise and farther-away distance measurements of the surrounding environments. The complementary information from these two sensors makes the two-modality fusion be a desired option. However, two major issues of the fusion between camera and LiDAR hinder its performance, i.e., how to effectively fuse these two modalities and how to precisely align them (suffering from the weak spatiotemporal synchronization problem). In this paper, we propose a coarse-to-fine LiDAR and camera fusion-based network (termed as LIF-Seg) for LiDAR segmentation. For the first issue, unlike these previous works fusing the point cloud and image information in a one-to-one manner, the proposed method fully utilizes the contextual information of images and introduces a simple but effective early-fusion strategy. Second, due to the weak spatiotemporal synchronization problem, an offset rectification approach is designed to align these two-modality features. The cooperation of these two components leads to the success of the effective camera-LiDAR fusion. Experimental results on the nuScenes dataset show the superiority of the proposed LIF-Seg over existing methods with a large margin. Ablation studies and analyses demonstrate that our proposed LIF-Seg can effectively tackle the weak spatiotemporal synchronization problem.

Scribble-Supervised LiDAR Semantic Segmentation

“…Fusion has been studied for a number of LiDAR based 3D perception tasks in a supervised and weakly-supervised manner [1,4,18,21,41,42]. For LiDAR semantic segmentation PMF [44] and LIF-Seg [40] fuse the information from streams that process each modality individually to obtain higher information yielding features. However such approaches not only require image information during inference but also have linearly increasing memory and computation cost.…”

Real-time multi-modal semantic fusion on unmanned aerial vehicles with label propagation for cross-domain adaptation

“…LIF-Seg by Zhao et al [24] improves upon the LiDAR segmentation network Cylinder3D [25] through early-and middle-fusion with color images. Image patches around the projected points provide per-point color context for earlyfusion, while mid-fusion concatenates semantic features from LiDAR and image, processed with Cylinder3D and DeepLab v3+, respectively, before processing with an additional refinement sub-network based on Cylinder3D for final semantic labels.…”

Grounding 3D Object Affordance from 2D Interactions in Images

“…It is aimed at establishing correspondences between instances from different modalities [3], either spatial or semantic. To achieve this, many methods use camera intrinsics to correspond spatial position of pixels and points, then align per pixel-point feature or fuse the raw [13,34,47] Image/Video 2D Heatmap Grounding [11,44,69] Point Cloud 3D Heatmap Grounding data [25,62,65,71,77,80]. Some works utilize depth information to project image features into 3D space and then fuse them with point-wise features [20,42,73,75,76].…”