BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Li, Zhiqi; Wang, Wenhai; Li, Hongyang; Xie, Enze; Sima, Chonghao; Lu, Tiecheng; Qiao, Yu; Dai, Jifeng

doi:10.48550/arxiv.2203.17270

Cited by 28 publications

(117 citation statements)

References 46 publications

Supporting

Mentioning

115

Contrasting

Order By: Relevance

“…Lift-Splat [23] learns categorical depth distributions in an unsupervised manner to generates the bird's eye view representations. With the trend of transformer, BEVFormer [13] learning bird's eye view representation from multi-camera images via spatiotemporal transformers to capture information across cameras. Bird's eye view based method which is based on depth information is sensitive to the accuracy of depth information.…”

Section: D Intermedia Representation Methodsmentioning

confidence: 99%

Consistency of Implicit and Explicit Features Matters for Monocular 3D Object Detection

Ye¹,

Jiang²,

Du³

2022

Preprint

View full text Add to dashboard Cite

Monocular 3D object detection is a common solution for low-cost autonomous agents to perceive their surrounding environment. Monocular detection has progressed into two categories: (1)Direct methods that infer 3D bounding boxes directly from a frontal-view image; (2)3D intermedia representation methods that map image features to 3D space for subsequent 3D detection. The second category is standing out not only because 3D detection forges ahead at the mercy of more meaningful and representative features, but because of emerging SOTA end-to-end prediction and planning paradigms that require a bird's-eye-view feature map from a perception pipeline. However, in transforming to 3D representation, these methods do not guarantee that objects' implicit orientations and locations in latent space are consistent with those explicitly observed in Euclidean space, which will hurt model performance. Hence, we argue that consistency of implicit and explicit features matters and present a novel monocular detection method, named CIEF, with the first orientation-aware image backbone to eliminate the disparity of implicit and explicit features in subsequent 3D representation. As a second contribution, we introduce a ray attention mechanism. In contrast to previous methods that repeat features along the projection ray or rely on another intermedia frustum point cloud, we directly transform image features to voxel representations with well-localized features. We also propose a handcrafted gaussian positional encoding function that outperforms the sinusoidal encoding function but maintains the benefit of being continuous. CIEF ranked 1st among all reported methods on both 3D and BEV detection benchmark of KITTI at submission time.

show abstract

Section: D Intermedia Representation Methodsmentioning

confidence: 99%

Consistency of Implicit and Explicit Features Matters for Monocular 3D Object Detection

Ye¹,

Jiang²,

Du³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…As in common autonomous driving datasets, the objects in general move on flat ground, PointPillars [19] proposes to map the 3D features onto a bird's eye view 2D space to reduce the computational overhead. It soon becomes a de-facto standard in this domain [37,19,15,52,31,22]. Lift-Splat-Shoot (LSS) [32] uses depth estimation network to extract the implied depth information of multiperspective images and transform camera feature maps into 3D Ego-car coordinate.…”

Section: Related Workmentioning

confidence: 99%

“…In the early stage of perception system, people design separate deep models for each sensor [35,36,56,15,51], and fusing the information via post-processing approaches [30]. Note that, people discover that bird's eye view (BEV) has been an de-facto standard for autonomous driving scenarios as, generally speaking, car cannot fly [19,22,37,15,52,31]. However, it is often difficult to regress 3D bounding boxes on pure image inputs due to the lack of depth information, and similarly, it is difficult to classify objects on point clouds when LiDAR does not receive enough points.…”

Section: Introductionmentioning

confidence: 99%

BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework

Liang¹,

Xie²,

Yu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Fusing the camera and LiDAR information has become a de-facto standard for 3D object detection tasks. Current methods rely on point clouds from the LiDAR sensor as queries to leverage the feature from the image space. However, people discover that this underlying assumption makes the current fusion framework infeasible to produce any prediction when there is a LiDAR malfunction, regardless of minor or major. This fundamentally limits the deployment capability to realistic autonomous driving scenarios. In contrast, we propose a surprisingly simple yet novel fusion framework, dubbed BEVFusion, whose camera stream does not depend on the input of LiDAR data, thus addressing the downside of previous methods. We empirically show that our framework surpasses the state-of-the-art methods under the normal training settings. Under the robustness training settings that simulate various LiDAR malfunctions, our framework significantly surpasses the state-of-the-art methods by 15.7% to 28.9% mAP. To the best of our knowledge, we are the first to handle realistic LiDAR malfunction and can be deployed to realistic scenarios without any post-processing procedure. The code is available at https://github.com/ADLab-AutoDrive/BEVFusion. Recently, people have designed LiDAR-camera fusion deep networks to better leverage information from both modalities. Specifically, the majority of works can be summarized as follow: i) given one or a few points of the LiDAR point cloud, LiDAR to world transformation matrix and the essential matrix (camera to world); ii) people transform the LiDAR points [41,44,45,44,16,57] or proposals § Corresponding Author.* Equal Contribution.Preprint. Under review.

show abstract

“…3D detection has received extensive attention as one of the fundamental tasks in autonomous driving scenarios [41,30,42,29,11,14,27,7,39,22]. Recently, fusing the two common modalities, input from the camera and LiDAR sensors, has become a de-facto standard in the 3D detection domain as each modality has complementary information of the other [4,33,34,31,40,8,44].…”

Section: Introductionmentioning

confidence: 99%

Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection

Yu¹,

Tang²,

Xie³

et al. 2022

Preprint

View full text Add to dashboard Cite

There are two critical sensors for 3D perception in autonomous driving, the camera and the LiDAR. The camera provides rich semantic information such as color, texture, and the LiDAR reflects the 3D shape and locations of surrounding objects. People discover that fusing these two modalities can significantly boost the performance of 3D perception models as each modality has complementary information to the other. However, we observe that current datasets are captured from expensive vehicles that are explicitly designed for data collection purposes, and cannot truly reflect the realistic data distribution due to various reasons. To this end, we collect a series of real-world cases with noisy data distribution, and systematically formulate a robustness benchmark toolkit, that simulates these cases on any clean autonomous driving datasets. We showcase the effectiveness of our toolkit by establishing the robustness benchmark on two widely-adopted autonomous driving datasets, nuScenes and Waymo, then, to the best of our knowledge, holistically benchmark the state-of-the-art fusion methods for the first time. We observe that: i) most fusion methods, when solely developed on these data, tend to fail inevitably when there is a disruption to the LiDAR input; ii) the improvement of the camera input is significantly inferior to the LiDAR one. We further propose an efficient robust training strategy to improve the robustness of the current fusion method. The benchmark and code are available at https://github.com/kcyu2014/lidar-camerarobust-benchmark.

show abstract

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Cited by 28 publications

References 46 publications

Consistency of Implicit and Explicit Features Matters for Monocular 3D Object Detection

Consistency of Implicit and Explicit Features Matters for Monocular 3D Object Detection

BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework

Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection

Contact Info

Product

Resources

About