Object as Query: Equipping Any 2D Object Detector with 3D Detection Ability

Abstract

3D object detection from multi-view images has drawn much attention over the past few years. Existing methods mainly establish 3D representations from multi-view images and adopt a dense detection head for object detection, or employ object queries distributed in 3D space to localize objects. In this paper, we design Multi-View 2D Objects guided 3D Object Detector (MV2D), which can be equipped with any 2D object detector to promote multi-view 3D object detection. Since 2D detections can provide valuable priors for object existence, MV2D exploits 2D detector to generate object queries conditioned on the rich image semantics. These dynamically generated queries enable MV2D to detect objects in larger 3D space without increased computational cost and show a strong capability of localizing 3D objects. For the generated queries, we design a sparse cross attention module to force them to focus on the features of specific objects, which reduces the computational cost and suppresses interference from noises. The evaluation results on the nuScenes dataset demonstrate the dynamic object queries and sparse feature aggregation do not harm 3D detection capability. MV2D also exhibits a state-of-the-art performance among existing methods. We hope MV2D can serve as a new baseline for future research.

Far3D: Expanding the Horizon for Surround-View 3D Object Detection

“…The results are barely satisfactory and we suppose that is because of the greater difficulty of depth estimation. We also reproduce MV2D (Wang et al 2023b) but it can hardly converge here. The reason is mainly the generated anchors lack accurate depth estimation, leading to large localization deviations over long distances.…”

“…3D detection from surround-view images can be improved through 2D auxiliary tasks, and some works (Xie et al 2022;Zhang et al 2023;Wang, Jiang, and Li 2022;Yang et al 2023;Wang et al 2023b) aim to exploit its potential. There are several approaches including 2D pertaining, auxiliary supervision, and proposal generation.…”

“…BEVFormerV2 (Yang et al 2023) presents a two-stage BEV detector where perspective proposals are fed into the BEV head for final predictions. MV2D (Wang et al 2023b) designs a 3D detector head that is initialized by RoI regions of 2D predicted proposals.…”

“…Motivated by this, leveraging high-quality 2D object priors to improve 3D proposals is a promising approach, for enabling accurate localization and comprehensive coverage. Although previous methods like Sim-MOD (Zhang et al 2023) and MV2D (Wang et al 2023b) have explored using 2D predictions to initialize 3D object proposals, they primarily focus on close-range tasks and discard learnable object queries. Moreover, as depicted in Fig.…”

Instance-Aware Multi-Camera 3D Object Detection with Structural Priors Mining and Self-Boosting Learning

“…As demonstrated in Fig 1(c), cars on the left column contain more precise textures and shape details versus those on the right column, thus reducing the ambiguity of the challenging depth estimation. Although some approaches (Chu et al 2023;Wang et al 2023) have explored 2D object priors for 3D object detection, they primarily leverage detected 2D objects after the perspective projection, thereby ignoring their potential to improve depth estimation for enhanced BEV feature construction.…”