BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers

Li, Zhiqi; Wang, Wenhai; Li, Hongyang; Xie, Enze; Sima, Chonghao; Lu, Tong; Qiao, Yu; Dai, Jifeng

doi:10.1007/978-3-031-20077-9_1

Cited by 553 publications

(179 citation statements)

References 41 publications

Supporting

Mentioning

176

Contrasting

Unclassified

Order By: Relevance

“…The significant improvement on mAP suggests MV2D has a strong capability at localizing objects in 3D space. In comparison with multi-view 3D object detection methods, MV2D with ResNet-101 outperforms BEV based methods BEVFormer-S [20] by 3.1% and 0.3% on mAP and NDS, and outperforms BEVDepth [19] with extra depth supervision by 3.0% and 4.3% on mAP and NDS. In comparison with query based methods, MV2D outperforms the best performing PETR [24] by 3.6% and 0.9% on mAP and NDS.…”

Section: Comparison With State-of-the-artsmentioning

confidence: 96%

“…ImVoxelNet [32] builds a 3D voxelized space and samples image features from multi-view to obtain the voxel representation. BEV-Former [20] leverages dense BEV queries to project and aggregate features from multi-view images by deformable attention [44]. BEVDet and BEVDepth [14,19] adopt the Lift-Splat module [29] to transform multi-view image features into the BEV representation based on the predicted depth distribution.…”

Section: Vision-based 3d Object Detectionmentioning

confidence: 99%

“…Moreover, adapting these methods to the multi-view setting relies on sophisticated cross- camera post-processing, which further causes degraded efficiency and efficacy. To handle these problems, recent researchers [14,19,20,24,39] propose to directly localize objects in 3D world space based on multi-view images, providing a new paradigm for vision-based 3D object detection.…”

Section: Introductionmentioning

confidence: 99%

“…According to the representation of fused features, current multi-view 3D object detection methods can be mainly divided into two streams: dense 3D methods and sparse query methods. Concretely, dense 3D methods render multi-view features into 3D space, such as Bird's-Eye-View (BEV) feature space [4,14,19,20] or voxel feature space [18,32]. However, since the computational costs are squarely proportional to the range of 3D space, they inevitably cannot scale up to large-scale scenarios [7].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Object as Query: Equipping Any 2D Object Detector with 3D Detection Ability

Wang¹,

Huang²,

Fu³

et al. 2023

Preprint

View full text Add to dashboard Cite

3D object detection from multi-view images has drawn much attention over the past few years. Existing methods mainly establish 3D representations from multi-view images and adopt a dense detection head for object detection, or employ object queries distributed in 3D space to localize objects. In this paper, we design Multi-View 2D Objects guided 3D Object Detector (MV2D), which can be equipped with any 2D object detector to promote multi-view 3D object detection. Since 2D detections can provide valuable priors for object existence, MV2D exploits 2D detector to generate object queries conditioned on the rich image semantics. These dynamically generated queries enable MV2D to detect objects in larger 3D space without increased computational cost and show a strong capability of localizing 3D objects. For the generated queries, we design a sparse cross attention module to force them to focus on the features of specific objects, which reduces the computational cost and suppresses interference from noises. The evaluation results on the nuScenes dataset demonstrate the dynamic object queries and sparse feature aggregation do not harm 3D detection capability. MV2D also exhibits a state-of-the-art performance among existing methods. We hope MV2D can serve as a new baseline for future research.

show abstract

Section: Comparison With State-of-the-artsmentioning

confidence: 96%

Section: Vision-based 3d Object Detectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Object as Query: Equipping Any 2D Object Detector with 3D Detection Ability

Wang¹,

Huang²,

Fu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…M2BEV [64] also investigates the viability of simultaneously running multi-tasks perception based on BEV features. BEVFormer [65] proposes a spatiotemporal transformer that aggregates BEV features from current and previous features via deformable attention [66]. Compared to object detection, semantic scene completion can provide occupancy for each small cell instead of assigning a fixed-size bounding box to an object.…”

Section: Related Workmentioning

confidence: 99%

VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion

Li¹,

Yu²,

Choy³

et al. 2023

Preprint

View full text Add to dashboard Cite

Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and understanding. To enable such capability in AI systems, we propose VoxFormer, a Transformerbased semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images. Our framework adopts a two-stage design where we start from a sparse set of visible and occupied voxel queries from depth estimation, followed by a densification stage that generates dense 3D voxels from the sparse ones. A key idea of this design is that the visual features on 2D images correspond only to the visible scene structures rather than the occluded or empty spaces. Therefore, starting with the featurization and prediction of the visible structures is more reliable. Once we obtain the set of sparse queries, we apply a masked autoencoder design to propagate the information to all the voxels by self-attention. Experiments on SemanticKITTI show that VoxFormer outperforms the state of the art with a relative improvement of 20.0% in geometry and 18.1% in semantics and reduces GPU memory during training by ∼45% to less than 16GB. Our code is available on https://github.com/NVlabs/VoxFormer.

show abstract

Multi-view LiDAR Guided Monocular 3D Object Detection

Liu

2022

Pattern Recognition and Computer Vision

View full text Add to dashboard Cite

BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers

Cited by 553 publications

References 41 publications

Object as Query: Equipping Any 2D Object Detector with 3D Detection Ability

Object as Query: Equipping Any 2D Object Detector with 3D Detection Ability

VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion

Multi-view LiDAR Guided Monocular 3D Object Detection

Contact Info

Product

Resources

About