Robust Fusion of LiDAR and Wide-Angle Camera Data for Autonomous Mobile Robots

Abstract

Autonomous robots that assist humans in day to day living tasks are becoming increasingly popular. Autonomous mobile robots operate by sensing and perceiving their surrounding environment to make accurate driving decisions. A combination of several different sensors such as LiDAR, radar, ultrasound sensors and cameras are utilized to sense the surrounding environment of autonomous vehicles. These heterogeneous sensors simultaneously capture various physical attributes of the environment. Such multimodality and redundancy of sensing need to be positively utilized for reliable and consistent perception of the environment through sensor data fusion. However, these multimodal sensor data streams are different from each other in many ways, such as temporal and spatial resolution, data format, and geometric alignment. For the subsequent perception algorithms to utilize the diversity offered by multimodal sensing, the data streams need to be spatially, geometrically and temporally aligned with each other. In this paper, we address the problem of fusing the outputs of a Light Detection and Ranging (LiDAR) scanner and a wide-angle monocular image sensor for free space detection. The outputs of LiDAR scanner and the image sensor are of different spatial resolutions and need to be aligned with each other. A geometrical model is used to spatially align the two sensor outputs, followed by a Gaussian Process (GP) regression-based resolution matching algorithm to interpolate the missing data with quantifiable uncertainty. The results indicate that the proposed sensor data fusion framework significantly aids the subsequent perception steps, as illustrated by the performance improvement of a uncertainty aware free space detection algorithm.

Survey of Datafusion Techniques for Laser and Vision Based Sensor Integration for Autonomous Navigation

“…A 360°camera captures dual images or video files from dual lenses with 180°field of view and either performs an on-camera automatic stitch of the images/video or lets the user perform off-board stitching of the images, to give a full 360°view of the world [28,[116][117][118].…”

“…The fusion of raw data in the first layer, a fusion of features in the second, and finally the decision layer fusion. In the case of the LiDAR and camera data fusion, two distinct steps effectively integrate/fuse the data [28,117,125]:…”

Real-Time Vehicle Detection Framework Based on the Fusion of LiDAR and Camera

“…After the pre-processing work is completed, the sparse depth map is transformed into a dense depth map through the depth completion framework so that the resolution of LiDAR data and image is the same. The depth-completion method can be divided into two types, guided depth completion [22,[26][27][28][29] and non-guided depth completion [21,30].…”

PCM: A Positioning Calibration Mechanism for Connected Autonomous Vehicles

“…In automatic driving scene, vehicles need to obtain not only their own accurate position information but also position information of other surrounding vehicles to furtherly assure safety [9]. In response to this, autonomous vehicles should be equipped with laser radar, millimeter-wave radar, binocular camera, and other sensing devices to detect surroundings, yet such devices are vulnerable to environmental interference, which could result in enormous error [10], [11]. The key to solving this problem is to connect autonomous vehicles to the network and introduce Road Side Unit (RSU) to automatic driving roads [12], [13].…”