DSGN++: Exploiting Visual-Spatial Relation for Stereo-based 3D Detectors

Abstract

Camera-based 3D object detectors are welcome due to their wider deployment and lower price than LiDAR sensors. We revisit the prior stereo modeling DSGN about the stereo volume constructions for representing both 3D geometry and semantics. We polish the stereo modeling and propose our approach, DSGN++, aiming for improving information flow throughout the 2D-to-3D pipeline in the following three main aspects. First, to effectively lift the 2D information to stereo volume, we propose depth-wise plane sweeping (DPS) that allows denser connections and extracts depth-guided features. Second, for better grasping differently spaced features, we present a novel stereo volume -Dual-view Stereo Volume (DSV) that integrates front-view and top-view features and reconstructs sub-voxel depth in the camera frustum. Third, as the foreground region becomes less dominant in 3D space, we firstly propose a multi-modal data editing strategy -Stereo-LiDAR Copy-Paste, which ensures cross-modal alignment and improves data efficiency. Without bells and whistles, extensive experiments in various modality setups on the popular KITTI benchmark show that our method consistently outperforms other camera-based 3D detectors for all categories. Code will be released at https://github.com/chenyilun95/DSGN2.

Application of deep learning for enhancing simultaneous localization and mapping in autonomous driving

“…Difference defined cost volume In computation, this algorithm will iterate over the D (maximum disparity) dimension. the cost volume here can be considered as the similarity between left and right cameras in disparity i, as disp i (x, y) = lef t(x, y) − right(x + i, y), like DSGN [18] and DSGN++ [17].…”

“…DSGN++ [17] is improved from DSGN [18] This allows the voxel to aggregate differently spaced 3D structure information and further expand the 2D-to-3D information flow. To perceive accurate front-surface depths, the depth head on stereo volume in 3D space (DSV) is first transformed to the frustum space followed by front-view depth supervision.…”