Velodyne LiDAR and monocular camera data fusion for depth map and 3D reconstruction

Akhtar, Mehreen; Qin, Huabiao; Chen, Guancheng

doi:10.1117/12.2539863

Cited by 5 publications

(9 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Depth cameras provide rich depth information, but their field of view is quite narrow. Conversely, LiDARs contain a wider field of view, but they provide sparse rather than rich environment information Akhtar et al (2019). LiDARs provide information in the form of a point cloud, whereas depth cameras provide luminance.…”

Section: Perception Sensingmentioning

confidence: 99%

A Novel Machine-Learning Framework With a Moving Platform for Maritime Drift Calculations

Bhaganagar¹,

Kolar

Faruqui

et al. 2022

Front. Mar. Sci.

View full text Add to dashboard Cite

A novel data-driven conceptual framework using a moving platform was developed to accurately estimate the drift of objects in the marine environment in real time using a combination of a perception-based sensing technology and deep-learning algorithms. The framework for conducting field experiments to establish the drift properties of moving objects is described. The objective of this study was to develop and test the integrated technology and determine the leeway drift characteristics of a full-scale three-dimensional mannequin resembling a person in water (PIW) and of a rectangular pelican box to accurately forecast the trajectory and the drift characteristics of the moving objects in real time. The wind and ocean current speeds were measured locally for the entire duration of the tests. A sensor hardware platform with a light detector and ranging sensor (LiDAR), stereoscopic depth cameras, a global positioning system, an inertial measurement unit, and operating software was designed and constructed by the team. It was then mounted on a boat (mobile test platform) to collect data. Tests were conducted by deploying the drifting objects from the mobile test platform into Galveston Bay and tracking them in real time. A framework was developed for applying machine learning and localization concepts on the data obtained from the sensors to determine the leeway trajectory, drift velocity, and leeway coefficients of the drifting objects in real time. Consistent trends in the downwind and crosswind leeway drift coefficients were observed for the pelican (significantly influenced by the wind) and PIW (influenced by the winds and currents).

show abstract

Section: Perception Sensingmentioning

confidence: 99%

A Novel Machine-Learning Framework With a Moving Platform for Maritime Drift Calculations

Bhaganagar¹,

Kolar

Faruqui

et al. 2022

Front. Mar. Sci.

View full text Add to dashboard Cite

show abstract

“…As recent as 2019, Akhtar et al [254] developed a data fusion system that was used to create a 3D Model with a depth map and object 3D reconstruction. Jin et al [255] proposed an approach for SLAM using 2D LiDAR and stereo camera with loop closures to estimate odometry.…”

Section: Mappingmentioning

confidence: 99%

Survey of Datafusion Techniques for Laser and Vision Based Sensor Integration for Autonomous Navigation

Kolar

Benavidez

Jamshidi

2020

Sensors

View full text Add to dashboard Cite

This paper focuses on data fusion, which is fundamental to one of the most important modules in any autonomous system: perception. Over the past decade, there has been a surge in the usage of smart/autonomous mobility systems. Such systems can be used in various areas of life like safe mobility for the disabled, senior citizens, and so on and are dependent on accurate sensor information in order to function optimally. This information may be from a single sensor or a suite of sensors with the same or different modalities. We review various types of sensors, their data, and the need for fusion of the data with each other to output the best data for the task at hand, which in this case is autonomous navigation. In order to obtain such accurate data, we need to have optimal technology to read the sensor data, process the data, eliminate or at least reduce the noise and then use the data for the required tasks. We present a survey of the current data processing techniques that implement data fusion using different sensors like LiDAR that use light scan technology, stereo/depth cameras, Red Green Blue monocular (RGB) and Time-of-flight (TOF) cameras that use optical technology and review the efficiency of using fused data from multiple sensors rather than a single sensor in autonomous navigation tasks like mapping, obstacle detection, and avoidance or localization. This survey will provide sensor information to researchers who intend to accomplish the task of motion control of a robot and detail the use of LiDAR and cameras to accomplish robot navigation.

show abstract

“…The projection of transformed points onto a 2D raster image is often used in multimodal image systems for data fusion resp. registration [ 9 , 18 , 19 ] ( Figure 3 , left). The challenges that arise in image data fusion [ 20 ] are: non-commensurability, different resolutions (challenge (B)), number of dimensions, noise, missing data, and conflicting, contradicting, or inconsistent data (challenge (D)).…”

Section: Introductionmentioning

confidence: 99%

“…Sensors such as Orbbec3D Astra Pro or Azure Kinect provide color point clouds at 30 fps, which is of particular interest for human–robot collaboration [ 22 , 23 ]. The fusion of camera data and modern 3D Light Detection and Ranging (LiDAR) data has many applications, including autonomous and safe control of mobile robots [ 18 ], object detection [ 8 ], and simultaneous localization and mapping (SLAM) [ 24 ]. Depth sensors are also widely used in scene analysis and human–computer interaction, e.g., object detection [ 8 , 17 , 25 ], (semantic) segmentation, [ 15 , 16 , 17 ] or markerless motion capture applications [ 26 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image

Junger

Buch

Notni

2023

Sensors

View full text Add to dashboard Cite

The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.

show abstract

Velodyne LiDAR and monocular camera data fusion for depth map and 3D reconstruction

Cited by 5 publications

References 6 publications

A Novel Machine-Learning Framework With a Moving Platform for Maritime Drift Calculations

A Novel Machine-Learning Framework With a Moving Platform for Maritime Drift Calculations

Survey of Datafusion Techniques for Laser and Vision Based Sensor Integration for Autonomous Navigation

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image

Contact Info

Product

Resources

About