3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization

Wang, Tsun-Hsuan; Hu, Hanwen; Lin, Chieh Hubert; Tsai, Yi-Hsuan; Chiu, Wei-Chen; Sun, Min

doi:10.1109/iros40897.2019.8968170

Cited by 37 publications

(51 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Currently, we implemented and tested our framework using the KITTI dataset (Geiger (2012)), and achieve a root mean squared error (RMSE) of 916.3 mm, which is better than the baseline (Kendall et al (2017)), and close to the state-of-art (Wang et al (2019)). In the future, we plan to improve our framework by adding more convolutional layers to our depth prediction network, implementing better cost functions to find the local optimum more accurately, and designing better refinement methods that take the disparities of neighbor points into consideration…”

Section: Current Results and Future Workmentioning

confidence: 90%

“…Previously, the general idea for this task is to plug LiDAR data into existing stereo matching algorithms. For example, Wang et al (2019) enhance the GC-Net (Kendall et al (2017)) by extracting more features and regularizing the cost volume using LiDAR data, while Park, Kim, and Sohn (2018) design a deep learning network to refine the output of SGM (Hirschmuller (2008)) with the help of LiDAR data. However, we argue that these stereo matching-base algorithms have two intrinsic drawbacks: first, intuitively, Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Re-Thinking LiDAR-Stereo Fusion Frameworks (Student Abstract)

Jin¹,

Duggirala²

2020

AAAI

View full text Add to dashboard Cite

In this paper, we present a 2-step framework for high-precision dense depth perception from stereo RGB images and sparse LiDAR input. In the first step, we train a deep neural network to predict dense depth map from the left image and sparse LiDAR data, in a novel self-supervised manner. Then in the second step, we compute a disparity map from the predicted depths, and refining the disparity map by making sure that for every pixel in the left, its match in the right image, according to the final disparity, is the local optimum.

show abstract

Section: Current Results and Future Workmentioning

confidence: 90%

Section: Introductionmentioning

confidence: 99%

Re-Thinking LiDAR-Stereo Fusion Frameworks (Student Abstract)

Jin¹,

Duggirala²

2020

AAAI

View full text Add to dashboard Cite

show abstract

“…There have been very few attempts to fuse LiDAR and stereo in autonomous driving. Previous methods [16,17,18] were designed for the depth completion and, as far as we know, there is no fusion involved for 3D object detection benchmark dataset. Cheng et al The approach proposed here is specifically intended for the depth completion task and tested on a 3D object detection task.…”

Section: Related Workmentioning

confidence: 99%

“…To enrich the representation for a normal stereo (RGBs) matching network, a decision has been taken to join the geometry information from the LiDAR point cloud. However, instead of directly using a 3D point cloud from LiDAR, like in [18], the 4-beam LiDAR point cloud is reprojected to both left and right image coordinates using the calibration parameters to obtain the two sparse 4-beam LiDAR depth maps corresponding to stereo images. Unlike [18] that used a simple early fusion paradigm by concating stereo images with their corresponding sparse LiDAR depth maps, the proposed method uses a late fusion approach, presented in the following section.…”

Section: Inputmentioning

confidence: 99%

“…They then proposed an unsupervised learning method to predict the dense depth map by fusing LiDAR and stereo which guides noise filtering in LiDAR point cloud. Wang et al[18] extracted features from LiDAR and stereo images and proposed the Conditional Cost Volume Normalization. The main aim is to adaptively regularize cost volume optimization based on LiDAR signals.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimation and 3D Object Detection

Minh¹,

Khoudour²,

Crouzil³

et al. 2021

IET Conf. Proc.

View full text Add to dashboard Cite

The ability to accurately detect and localize objects is recognized as being the most important for the perception of selfdriving cars. From 2D to 3D object detection, the most difficult is to determine the distance from the ego-vehicle to objects. Expensive technology like LiDAR can provide a precise and accurate depth information, so most studies have tended to focus on this sensor showing a performance gap between LiDAR-based methods and camera-based methods. Although many authors have investigated how to fuse LiDAR with RGB cameras, as far as we know there are no studies to fuse LiDAR and stereo in a deep neural network for the 3D object detection task. This paper presents SLS-Fusion, a new approach to fuse data from 4-beam LiDAR and a stereo camera via a neural network for depth estimation to achieve better dense depth maps and thereby improves 3D object detection performance. Since 4-beam LiDAR is cheaper than the well-known 64-beam LiDAR, this approach is also classified as a low-cost sensorsbased method. Through evaluation on the KITTI benchmark, it is shown that the proposed method significantly improves depth estimation performance compared to a baseline method. Also when applying it to 3D object detection, a new state of the art on low-cost sensor based method is achieved.

show abstract

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

Meng

Li²,

Zhong

et al. 2023

Journal of Field Robotics

View full text Add to dashboard Cite

Light detection and ranging (LiDAR) and stereo cameras are two generally used solutions for perceiving 3D information. The complementary properties of these two sensor modalities motivate a fusion to derive practicable depth sensing toward real‐world applications. Promoted by deep neural network (DNN) techniques, recent works achieve superior performance on accuracy. However, the complex architecture and the sheer number of DNN parameters often lead to poor generalization capacity and non‐real‐time computing. In this paper, we present FastFusion, a three‐stage stereo‐LiDAR deep fusion scheme, which integrates the LiDAR priors into each step of classical stereo‐matching taxonomy, gaining high‐precision dense depth sensing in a real‐time manner. We integrate stereo‐LiDAR information by taking advantage of a compact binary neural network and utilize the proposed cross‐based LiDAR trust aggregation to further fuse the sparse LiDAR measurements in the back‐end of stereo matching. To align the photometrical of the input image and the depth of the estimation, we introduce a refinement network to guarantee consistency. More importantly, we present a graphic processing unit‐based acceleration framework for providing a low‐latency implementation of FastFusion, gaining both accuracy improvement and real‐time responsiveness. In the experiments, we demonstrate the effectiveness and practicability of FastFusion, which obtains a significant speedup over state‐of‐the‐art baselines while achieving comparable accuracy on depth sensing. The video demo for real‐time depth estimation of FastFusion on the real‐world driving scenario is available at https://youtu.be/nP7cls2BA8s.

show abstract

3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization

Cited by 37 publications

References 25 publications

Re-Thinking LiDAR-Stereo Fusion Frameworks (Student Abstract)

Re-Thinking LiDAR-Stereo Fusion Frameworks (Student Abstract)

Sparse LiDAR and Stereo Fusion (SLS-Fusion) for Depth Estimation and 3D Object Detection

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

Contact Info

Product

Resources

About