LiDAR-based Recurrent 3D Semantic Segmentation with Temporal Memory Alignment

Duerr, Fabian; Pfaller, Mario; Weigel, Hendrik; Beyerer, Jürgen

doi:10.1109/3dv50981.2020.00088

Cited by 31 publications

(13 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the quantitative evaluation, we compare our proposed approach against other methods on the multiple scans task in semantic segmentation benchmark. As shown in Table I, our presented Meta-RangeSeg method outperforms the stateof-the-art approach [26] by 2.5% on SemanticKITTI test set, which also achieves the best results on the most of classes. It is worthy of mentioning that our approach can process the input point cloud data at a rate of 22Hz while maintaining high accuracy.…”

Section: Performance Evaluationmentioning

confidence: 81%

“…Shi et al [14] employ a voxel-based 3D sparse convolutional network to fuse local information from the previous and current frames through local interpolation, which only make use of the two consecutive scans. Duerr et al [26] propose a recurrent segmentation architecture using range images, which recursively aggregate the features of previous scans in order to exploit the short term temporal dependencies. In [15], superimposing point clouds in 3D space is adopted for multiple scans segmentation, whose memory consumption and computational time increase linearly with the total number of scans per input model.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Meta-RangeSeg: LiDAR Sequence Semantic Segmentation Using Multiple Feature Aggregation

Wang¹,

Zhu²,

Zhang³

2022

Preprint

View full text Add to dashboard Cite

LiDAR sensor is essential to the perception system in autonomous vehicles and intelligent robots. To fulfill the realtime requirements in real-world applications, it is necessary to efficiently segment the LiDAR scans. Most of previous approaches directly project 3D point cloud onto the 2D spherical range image so that they can make use of the efficient 2D convolutional operations for image segmentation. Although having achieved the encouraging results, the neighborhood information is not wellpreserved in the spherical projection. Moreover, the temporal information is not taken into consideration in the single scan segmentation task. To tackle these problems, we propose a novel approach to semantic segmentation for LiDAR sequences named Meta-RangeSeg, where a novel range residual image representation is introduced to capture the spatial-temporal information. Specifically, Meta-Kernel is employed to extract the meta features, which reduces the inconsistency between the 2D range image coordinates input and Cartesian coordinates output. An efficient U-Net backbone is used to obtain the multi-scale features. Furthermore, Feature Aggregation Module (FAM) aggregates the meta features and multi-scale features, which tends to strengthen the role of range channel. We have conducted extensive experiments for performance evaluation on SemanticKITTI, which is the de-facto dataset for LiDAR semantic segmentation. The promising results show that our proposed Meta-RangeSeg method is more efficient and effective than the existing approaches. Our full implementation is publicly available at https://github.com/songw-zju/Meta-RangeSeg.

show abstract

Section: Performance Evaluationmentioning

confidence: 81%

Section: Related Workmentioning

confidence: 99%

Meta-RangeSeg: LiDAR Sequence Semantic Segmentation Using Multiple Feature Aggregation

Wang¹,

Zhu²,

Zhang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…PandaSet [29] provides 6080 point-wise annotated lidar scans of a Pandar64, with corresponding camera images from the front center camera. For comparability with other approaches [23] we group the labeled classes into a subset of 14 classes and follow the data split proposed in [23].…”

Section: Discussionmentioning

confidence: 99%

“…The lidar backbone computes features for the input point clouds, which are represented in range view, based on the spherical projection of [21], [23]. Its architecture is motivated by EfficientPS [8] and adapted to the range view.…”

Section: A Lidar Backbonementioning

confidence: 99%

Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation

Schieber¹,

Duerr²,

Schoen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Robust environment perception for autonomous vehicles is a tremendous challenge, which makes a diverse sensor set with e.g. camera, lidar and radar crucial. In the process of understanding the recorded sensor data, 3D semantic segmentation plays an important role. Therefore, this work presents a pyramid-based deep fusion architecture for lidar and camera to improve 3D semantic segmentation of traffic scenes. Individual sensor backbones extract feature maps of camera images and lidar point clouds. A novel Pyramid Fusion Backbone fuses these feature maps at different scales and combines the multimodal features in a feature pyramid to compute valuable multimodal, multi-scale features. The Pyramid Fusion Head aggregates these pyramid features and further refines them in a late fusion step, incorporating the final features of the sensor backbones. The approach is evaluated on two challenging outdoor datasets and different fusion strategies and setups are investigated. It outperforms recent range view based lidar approaches as well as all so far proposed fusion strategies and architectures.

show abstract

“…Previous works tackled this problem by extracting temporal information from residual range images [5] or bird's eye view (BEV) images [26], typically using a 2D convolutional neural network (CNN). The back-projection from these 2D representations to the 3D space often requires post-processing like k-nearest neighbor (kNN) clustering [5], [9], [12], [25] to avoid labels bleeding into points that are close in the image space but distant in 3D. Other approaches can identify objects that have moved in 3D space directly during mapping [1] or with a clustering and tracking approach [6].…”

Section: Non-moving Movingmentioning

confidence: 99%

Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions

Mersch

Chen

Vizzo

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

A key challenge for autonomous vehicles is to navigate in unseen dynamic environments. Separating moving objects from static ones is essential for navigation, pose estimation, and understanding how other traffic participants are likely to move in the near future. In this work, we tackle the problem of distinguishing 3D LiDAR points that belong to currently moving objects, like walking pedestrians or driving cars, from points that are obtained from non-moving objects, like walls but also parked cars. Our approach takes a sequence of observed LiDAR scans and turns them into a voxelized sparse 4D point cloud. We apply computationally efficient sparse 4D convolutions to jointly extract spatial and temporal features and predict moving object confidence scores for all points in the sequence. We develop a receding horizon strategy that allows us to predict moving objects online and to refine predictions on the go based on new observations. We use a binary Bayes filter to recursively integrate new predictions of a scan resulting in more robust estimation. We evaluate our approach on the SemanticKITTI moving object segmentation challenge and show more accurate predictions than existing methods. Since our approach only operates on the geometric information of point clouds over time, it generalizes well to new, unseen environments, which we evaluate on the Apollo dataset. Index Terms-Semantic Scene Understanding; Deep Learning Methods I. INTRODUCTION D ISTINGUISHING moving from static objects in 3D LiDAR data is a crucial task for autonomous systems and required for planning collision-free trajectories and navigating safely in dynamic environments. Moving object segmentation (MOS) can improve localization [5], [7], planning [34], mapping [5], scene flow estimation [2], [15], [37], or the prediction of future states [38], [40]. There are mapping approaches that identify if observed points are potentially moving or have moved throughout the mapping process [1], [7], [16], [28]. On the contrary, identifying objects that are actually moving within a short time horizon are of interest for online navigation [34], can improve scene flow estimation between two consecutive point clouds [2], [15], [37], or support predicting a future state of the environment [40].

show abstract

LiDAR-based Recurrent 3D Semantic Segmentation with Temporal Memory Alignment

Cited by 31 publications

References 21 publications

Meta-RangeSeg: LiDAR Sequence Semantic Segmentation Using Multiple Feature Aggregation

Meta-RangeSeg: LiDAR Sequence Semantic Segmentation Using Multiple Feature Aggregation

Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation

Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions

Contact Info

Product

Resources

About