Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-view Transformation

Yang, Weixiang; Li, Qi; Liu, Wenxi; Yu, Yuanlong; Ma, Yuexin; He, Shengfeng; Pan, Jia

doi:10.1109/cvpr46437.2021.01528

Cited by 84 publications

(84 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Methods [30,16,9] utilize multilayer perceptron to learn the translation from perspective view to the BEV. PYVA [50] proposes a cross-view transformer that converts the front-view monocular image into the BEV, but this paradigm is not suitable for fusing multi-camera features due to the computational cost of global attention mechinism [42]. In addition to the spatial information, previous works [18,38,6] also consider the temporal information by stacking BEV features from several timestamps.…”

Section: Camera-based 3d Perceptionmentioning

confidence: 99%

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Li¹,

Wang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

3D visual perception tasks, including 3D detection and map segmentation based on multi-camera images, are essential for autonomous driving systems. In this work, we present a new framework termed BEVFormer, which learns unified BEV representations with spatiotemporal transformers to support multiple autonomous driving perception tasks. In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries. To aggregate spatial information, we design a spatial cross-attention that each BEV query extracts the spatial features from the regions of interest across camera views. For temporal information, we propose a temporal self-attention to recurrently fuse the history BEV information. Our approach achieves the new state-of-the-art 56.9% in terms of NDS metric on the nuScenes test set, which is 9.0 points higher than previous best arts and on par with the performance of LiDAR-based baselines. We further show that BEVFormer remarkably improves the accuracy of velocity estimation and recall of objects under low visibility conditions. The code will be released at https://github.com/zhiqi-li/BEVFormer.

show abstract

Section: Camera-based 3d Perceptionmentioning

confidence: 99%

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Li¹,

Wang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, NEAT [26] uses transformer encoder and converts image features to BEV space with a MLP-based attention by traversing through all the grid in the BEV space. PYVA [27] generates BEV features from a CNN encoder-decoder structure by MLP. The feature is then enhanced by a transformer cross-attention module.…”

Section: Related Workmentioning

confidence: 99%

BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs

Peng¹,

Chen²,

Fu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Semantic segmentation in bird's eye view (BEV) is an important task for autonomous driving. Though this task has attracted a large amount of research efforts, it is still challenging to flexibly cope with arbitrary (single or multiple) camera sensors equipped on the autonomous vehicle. In this paper, we present BEVSegFormer, an effective transformerbased method for BEV semantic segmentation from arbitrary camera rigs. Specifically, our method first encodes image features from arbitrary cameras with a shared backbone. These image features are then enhanced by a deformable transformerbased encoder. Moreover, we introduce a BEV transformer decoder module to parse BEV semantic segmentation results. An efficient multi-camera deformable attention unit is designed to carry out the BEV-to-image view transformation. Finally, the queries are reshaped according the layout of grids in the BEV, and upsampled to produce the semantic segmentation result in a supervised manner. We evaluate the proposed algorithm on the public nuScenes dataset and a self-collected dataset. Experimental results show that our method achieves promising performance on BEV semantic segmentation from arbitrary camera rigs. We also demonstrate the effectiveness of each component via ablation study.

show abstract

“…In the cross-view domain, some novel and effective transformer structures have also been proposed to implement different downstream tasks.Chen et al [55] proposed a pair of cross-view transformers to transform the feature maps into the other view and introduce cross-view consistency loss on them. Yang et al [56] presented a novel framework that enables reconstructing a local map formed by road layout and vehicle occupancy in the bird's-eye view given a front-view monocular image only, and a cross-view transformation module was proposed to strengthen the view transformation and scene understanding. Tulder et al [57] presented a novel cross-view transformer method to transfer information between unregistered views at the level of spatial feature maps, which achieved remarkable results in field of Multi-view medical image analysis.…”

Section: B Transformer In Visionmentioning

confidence: 99%

A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization

Dai,

Hu,

Zhuang

et al. 2022

Preprint

View full text Add to dashboard Cite

Cross-view geo-localization is a task of matching the same geographic image from different views, e.g., unmanned aerial vehicle (UAV) and satellite. The most difficult challenges are the position shift and the uncertainty of distance and scale. Existing methods are mainly aimed at digging for more comprehensive fine-grained information. However, it underestimates the importance of extracting robust feature representation and the impact of feature alignment. The CNN-based methods have achieved great success in cross-view geo-localization. However it still has some limitations, e.g., it can only extract part of the information in the neighborhood and some scale reduction operations will make some fine-grained information lost. In particular, we introduce a simple and efficient transformer-based structure called Feature Segmentation and Region Alignment (FSRA) to enhance the model's ability to understand contextual information as well as to understand the distribution of instances. Without using additional supervisory information, FSRA divides regions based on the heat distribution of the transformer's feature map, and then aligns multiple specific regions in different views one on one. Finally, FSRA integrates each region into a set of feature representations. The difference is that FSRA does not divide regions manually, but automatically based on the heat distribution of the feature map. So that specific instances can still be divided and aligned when there are significant shifts and scale changes in the image. In addition, a multiple sampling strategy is proposed to overcome the disparity in the number of satellite images and that of images from other sources. Experiments show that the proposed method has superior performance and achieves the state-of-the-art in both tasks of drone view target localization and drone navigation. Code will be released at https://github.com/Dmmm1997/FSRA

show abstract

Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-view Transformation

Cited by 84 publications

References 40 publications

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs

A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization

Contact Info

Product

Resources

About