Noise-Aware Unsupervised Deep Lidar-Stereo Fusion

Abstract

In this paper, we present LidarStereoNet, the first unsupervised Lidar-stereo fusion network, which can be trained in an end-to-end manner without the need of ground truth depth maps. By introducing a novel "Feedback Loop" to connect the network input with output, LidarStereoNet could tackle both noisy Lidar points and misalignment between sensors that have been ignored in existing Lidarstereo fusion studies. Besides, we propose to incorporate a piecewise planar model into network learning to further constrain depths to conform to the underlying 3D geometry. Extensive quantitative and qualitative evaluations on both real and synthetic datasets demonstrate the superiority of our method, which outperforms state-of-the-art stereo matching, depth completion and Lidar-Stereo fusion approaches significantly.

Data Processing Approaches on SPAD-Based d-TOF LiDAR Systems: A Review

“…Therefore, algorithms that have been proven effective on KITTI dataset alone may not be generalizable to other datasets. Researchers propose semisupervised [61] or unsupervised [58] approaches to mitigate this problem. Besides, the development of the target simulators or complete virtual measurement environments can be an interesting topic.…”

“…Known problems include insufficient number of incoming photons due to low reflectivity, sparse spatial resolution, and large amounts of noise due to high background light intensity. These drawbacks can significantly degrade the performance of the post-processing [58].…”

“…In [60], an upsampling method based on a convolutional neural network (CNN) is presented dealing with the limited spatial resolution of LiDAR data. Cheng et al introduces a feedback loop to automatically clean the noise data in the PCD [58]. In addition, LiDAR data are used as a complement to range information in many sensor fusion applications.…”

“…In addition, LiDAR data are used as a complement to range information in many sensor fusion applications. The most common example is the fusion between the LiDAR data and the stereo images [58] [61] [62] [63]. The use of images to guide the reconstruction of depth information is collectively known as depth completion.…”

“…Through the concept of transfer learning [83] [84], the encoder-decoder network structure in [85], which shares an encoder while designs individual decoders for different tasks, can reduce the repetitive operations arising from simple combinations of algorithms. As reported in [58], even a small, five-layer network contains 25,000 parameters. Ren et al argue that the real-time performance in autonomous driving is as important as the inference accuracy and proposed their SBNet to speed up the training and the calculation in CNN [82].…”