Deep Generative Modeling of LiDAR Data

Caccia, Lucas; Hoof, Herke van; Courville, Aaron; Pineau, Joëlle

doi:10.1109/iros40897.2019.8968535

Cited by 39 publications

(29 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These deep networks have shown promising performances on object classification and scene understanding, but none have been applied to radar signal processing. Although there are studies on extracting points from RGB-D [41] or LiDAR scanners [42], constructing and analyzing points from a range-Doppler radar system are different concepts. Instead of directly processing the input points, we add an intensity-based reconstruction module before the hierarchical PointNet model, which incorporates the radar domain knowledge into the 3D network.…”

Section: Related Workmentioning

confidence: 99%

Human Behavior Recognition Using Range-Velocity-Time Points

Chen

2020

IEEE Access

View full text Add to dashboard Cite

Radar-based sensors do not require optimal lighting and atmospheric conditions and nonocclusion, making them a promising solution for human behavior analysis in complex environments. Existing radar-based models generally retrieve features from either the time-velocity domain or the time-range domain. Such two-dimensional representations cannot fully depict dynamic human motion features. In this paper, we propose a temporal range-Doppler PointNet-based method to analyze human behavior. We transform human echoes to 3D point sets and then feed them into the hierarchical PointNet model for classification. The proposed point network can learn structural features from the micromotion trajectory more effectively than directly processing the raw point cloud. To further improve our model's robustness in practical applications, we design an outlier detection module for detecting anomalies such as in multitarget scenarios. The results of experiments on motion capture databases and range-Doppler radar measurements demonstrate that our method realizes outstanding performance in terms of the classification accuracy, noise robustness, and anomaly detection accuracy. INDEX TERMS Human activity recognition, micro-Doppler effect, deep learning, point set, range-Doppler processing, ultra-wideband radar.

show abstract

Section: Related Workmentioning

confidence: 99%

Human Behavior Recognition Using Range-Velocity-Time Points

Chen

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…In semantic point segmentation [1,2] and object detection tasks [3], these approaches have gained performance in practice. As for GANs with LiDAR data, Caccia et al [10] trained a GAN generator with Cartesian points projected onto sensor-view 2D grids.…”

Section: Introductionmentioning

confidence: 99%

“…The goal of this study is to build a generative model of LiDAR data based on the 2D representation [1]- [3,10,11]. The existing work on GANs [10] relied on preprocessing LiDAR data such as using heuristics to fill in the pointdrops. Moreover, the quantitative performance has not yet been evaluated in terms of quality and diversity.…”

Section: Introductionmentioning

confidence: 99%

Learning to Drop Points for LiDAR Scan Synthesis

Nakashima¹,

Kurazume²

2021

Preprint

View full text Add to dashboard Cite

Generative modeling of 3D scenes is a crucial topic for aiding mobile robots to improve unreliable observations. However, despite the rapid progress in the natural image domain, building generative models is still challenging for 3D data, such as point clouds. Most existing studies on point clouds have focused on small and uniform-density data. In contrast, 3D LiDAR point clouds widely used in mobile robots are nontrivial to be handled because of the large number of points and varying-density. To circumvent this issue, 3D-to-2D projected representation such as a cylindrical depth map has been studied in existing LiDAR processing tasks but susceptible to discrete lossy pixels caused by failures of laser reflection. This paper proposes a novel framework based on generative adversarial networks to synthesize realistic LiDAR data as an improved 2D representation. Our generative architectures are designed to learn a distribution of inverse depth maps and simultaneously simulate the lossy pixels, which enables us to decompose an underlying smooth geometry and the corresponding uncertainty of laser reflection. To simulate the lossy pixels, we propose a differentiable framework to learn to produce sample-dependent binary masks using the Gumbel-Sigmoid reparametrization trick. We demonstrate the effectiveness of our approach in synthesis and reconstruction tasks on two LiDAR datasets. We further showcase potential applications by recovering various corruptions in LiDAR data.

show abstract

“…In order to leverage mature 2D detection models to extract visual cues from multiple signals, we first convert ambient and reflectance signals into 2D image representation called "LiDAR image". Unlike previous methods where point cloud is projected into image space with the calibration matrix, we re-organize the LiDAR multi-modal measurement into a 2D image according to the alignment of LiDAR detector array (i.e., range view[1]), as shown inFig 1. Specifically, each column of pixels are signals captured by the vertically aligned LiDAR detectors and each row of pixels are signals captured by the same detector in different horizontal directions.…”

mentioning

confidence: 99%

Multi-Echo LiDAR for 3D Object Detection

Man

Weng

Sivakumar

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

LiDAR sensors can be used to obtain a wide range of measurement signals other than a simple 3D point cloud, and those signals can be leveraged to improve perception tasks like 3D object detection. A single laser pulse can be partially reflected by multiple objects along its path, resulting in multiple measurements called echoes. Multi-echo measurement can provide information about object contours and semi-transparent surfaces which can be used to better identify and locate objects. LiDAR can also measure surface reflectance (intensity of laser pulse return), as well as ambient light of the scene (sunlight reflected by objects). These signals are already available in commercial LiDAR devices but have not been used in most LiDAR-based detection models. We present a 3D object detection model which leverages the full spectrum of measurement signals provided by LiDAR. First, we propose a multi-signal fusion (MSF) module to combine (1) the reflectance and ambient features extracted with a 2D CNN, and (2) point cloud features extracted using a 3D graph neural network (GNN). Second, we propose a multi-echo aggregation (MEA) module to combine the information encoded in different set of echo points. Compared with traditional single echo point cloud methods, our proposed Multi-Signal LiDAR Detector (MSLiD) extracts richer context information from a wider range of sensing measurements and achieves more accurate 3D object detection. Experiments show that by incorporating the multi-modality of LiDAR, our method outperforms the state-of-the-art by up to 9.1%.

show abstract

Deep Generative Modeling of LiDAR Data

Cited by 39 publications

References 37 publications

Human Behavior Recognition Using Range-Velocity-Time Points

Human Behavior Recognition Using Range-Velocity-Time Points

Learning to Drop Points for LiDAR Scan Synthesis

Multi-Echo LiDAR for 3D Object Detection

Contact Info

Product

Resources

About