PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving

Xiao, Pengchuan; Shao, Zhenlei; Sheng, Hao; Zhang, Zishuo; Chai, Xiaolin; Jiao, Judy; Li, Zesong; Wu, Jian; Sun, Kai; Jiang, Kun; Wang, Yunlong; Yang, Diange

doi:10.1109/itsc48978.2021.9565009

Cited by 134 publications

(31 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The following evaluation is conducted on two challenging and large-scale outdoor datatsets, SemanticKITTI [28] and PandaSet [29], and in both cases the mean Intersection over-Union (mIoU) of the overlapping field of view of both sensors is reported to investigate our approach and compare it to other state-of-the-art approaches. SemanticKITTI is a point-wise annotated dataset and based on 360 • lidar scans of a Velodyne-HDL64E from the odometry task of the KITTI Vision Benchmark [30].…”

Section: Discussionmentioning

confidence: 99%

Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation

Schieber¹,

Duerr²,

Schoen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Robust environment perception for autonomous vehicles is a tremendous challenge, which makes a diverse sensor set with e.g. camera, lidar and radar crucial. In the process of understanding the recorded sensor data, 3D semantic segmentation plays an important role. Therefore, this work presents a pyramid-based deep fusion architecture for lidar and camera to improve 3D semantic segmentation of traffic scenes. Individual sensor backbones extract feature maps of camera images and lidar point clouds. A novel Pyramid Fusion Backbone fuses these feature maps at different scales and combines the multimodal features in a feature pyramid to compute valuable multimodal, multi-scale features. The Pyramid Fusion Head aggregates these pyramid features and further refines them in a late fusion step, incorporating the final features of the sensor backbones. The approach is evaluated on two challenging outdoor datasets and different fusion strategies and setups are investigated. It outperforms recent range view based lidar approaches as well as all so far proposed fusion strategies and architectures.

show abstract

Section: Discussionmentioning

confidence: 99%

Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation

Schieber¹,

Duerr²,

Schoen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The second experiment involves high-resolution road scene images. ViCE is trained on a set of 1M varying road scene images collected from eight public datasets [40,55,57,78,88,102,109,116]. We evaluate performance on the Cityscapes dataset [25] ages like the baselines, demonstrating the versatility and domain generalization power of our method.…”

Section: Representation Quality Experimentsmentioning

confidence: 97%

ViCE: Visual Concept Embedding Discovery and Superpixelization

Karlsson¹,

Hayashi²,

Fujii³

et al. 2021

Preprint

View full text Add to dashboard Cite

This work presents a self-supervised method to learn dense semantically rich visual concept embeddings for images inspired by methods for learning word embeddings in NLP. Our method improves on prior work by generating more expressive embeddings and by being applicable for high-resolution images. Viewing the generation of natural images as a stochastic process where a set of latent visual concepts give rise to observable pixel appearances, our method is formulated to learn the inverse mapping from pixels to concepts. Our method greatly improves the effectiveness of self-supervised learning for dense embedding maps by introducing superpixelization as a natural hierarchical step up from pixels to a small set of visually coherent regions. Additional contributions are regional contextual masking with nonuniform shapes matching visually coherent patches and complexity-based view sampling inspired by masked language models. The enhanced expressiveness of our dense embeddings is demonstrated by significantly improving the state-of-the-art representation quality benchmarks on COCO (+12.94 mIoU, +87.6%) and Cityscapes (+16.52 mIoU, +134.2%). Results show favorable scaling and domain generalization properties not demonstrated by prior work.

show abstract

“…Therefore, we choose four datasets for our training split (i.e. Lyft L5 [44], Nuscenes [45], Pandaset [46], Ford AV Dataset [47]), one dataset for validating the training (i.e. Argoverse [48]) and report test results on two datasets (i.e.…”

Section: A Datamentioning

confidence: 99%

“…While modern autonomous driving datasets (e.g. [44], [45], [46], [47], [48]) provide cameras with full 360°fieldof-view, older datasets such as KITTI have only a frontfacing stereo camera. Since our method utilizes lidar points that are covered by the viewing frustrum of at least one on-board camera, the older KITTI dataset cannot utilize its full potential.…”

Section: A Datamentioning

confidence: 99%

Continuous Self-Localization on Aerial Images Using Visual and Lidar Sensors

Fervers¹,

Bullinger²,

Bodensteiner³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper proposes a novel method for geo-tracking, i.e. continuous metric self-localization in outdoor environments by registering a vehicle's sensor information with aerial imagery of an unseen target region. Geo-tracking methods offer the potential to supplant noisy signals from global navigation satellite systems (GNSS) and expensive and hard to maintain prior maps that are typically used for this purpose. The proposed geo-tracking method aligns data from on-board cameras and lidar sensors with geo-registered orthophotos to continuously localize a vehicle. We train a model in a metric learning setting to extract visual features from ground and aerial images. The ground features are projected into a top-down perspective via the lidar points and are matched with the aerial features to determine the relative pose between vehicle and orthophoto.Our method is the first to utilize on-board cameras in an end-to-end differentiable model for metric self-localization on unseen orthophotos. It exhibits strong generalization, is robust to changes in the environment and requires only geo-poses as ground truth. We evaluate our approach on the KITTI-360 dataset and achieve a mean absolute position error (APE) of 0.94m. We further compare with previous approaches on the KITTI odometry dataset and achieve state-of-the-art results on the geo-tracking task.

show abstract

PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving

Cited by 134 publications

References 22 publications

Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation

Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation

ViCE: Visual Concept Embedding Discovery and Superpixelization

Continuous Self-Localization on Aerial Images Using Visual and Lidar Sensors

Contact Info

Product

Resources

About