ST3D++: Denoised Self-training for Unsupervised Domain Adaptation on 3D Object Detection

Abstract

In this paper, we present a self-training method, named ST3D++, with a holistic pseudo label denoising pipeline for unsupervised domain adaptation on 3D object detection. ST3D++ aims at reducing noise in pseudo label generation as well as alleviating the negative impacts of noisy pseudo labels on model training. First, ST3D++ pre-trains the 3D object detector on the labeled source domain with random object scaling (ROS) which is designed to reduce target domain pseudo label noise arising from object scale bias of the source domain. Then, the detector is progressively improved through alternating between generating pseudo labels and training the object detector with pseudo-labeled target domain data. Here, we equip the pseudo label generation process with a hybrid quality-aware triplet memory to improve the quality and stability of generated pseudo labels. Meanwhile, in the model training stage, we propose a source data assisted training strategy and a curriculum data augmentation policy to effectively rectify noisy gradient directions and avoid model over-fitting to noisy pseudo labeled data. These specific designs enable the detector to be trained on meticulously refined pseudo labeled target data with denoised training signals, and thus effectively facilitate adapting an object detector to a target domain without requiring annotations. Finally, our method is assessed on four 3D benchmark datasets (i.e. Waymo, KITTI, Lyft, and nuScenes) for three common categories (i.e. car, pedestrian and bicycle). ST3D++ achieves state-of-the-art performance on all evaluated settings, outperforming the corresponding baseline by a large margin (e.g. 9.6% ∼ 38.16% on Waymo → KITTI in terms of AP 3D ), and even surpasses the fully supervised oracle results on the KITTI 3D object detection benchmark with target prior. Code will be available.

Voxel-MAE: Masked Autoencoders for Pre-training Large-scale Point Clouds

“…MLC-Net [14] leverages the mean-teacher paradigm with three levels of consistency to facilitate the cross-domain transfer. The self-denoising frameworks, ST3D [33] and ST3D++ [32], employ three strategies (i.e., random object scaling, hybrid quality-aware triplet memory, and curriculum data augmentation), to reduce noise in pseudo label generation. Though achieving promising results, most of the aforementioned domain adaptation methods ignore the target data distribution when training the model with source data.…”

Vision based 3D Object Detection using Deep Learning: Methods with Challenges and Applications towards Future Directions

“…Domain gaps for different datasets depend on object size, weather condition, specific locations, and orientation [57]. Overall gaps can be categorized in to two categories mentioned below.…”

“…2) nuScenes dataset: nuScenes dataset contains 1000 segments of 20 seconds each for 3D object detection where 750, 150 and 150 segments for training, validation and testing, respectively [10,68,69,57,70]. Annotation rate is 2Hz for which 28k, 6k and 6k annotated frames are available in these datasets for training, validation and testing respectively.…”

See Eye to Eye: A Lidar-Agnostic 3D Detection Framework for Unsupervised Multi-Target Domain Adaptation

“…Wang et al [56] proposed a semi-supervised approach using object-size statistics of the target domain to resize training samples in the labelled source domain. A popular approach is the use of self-training [43,63,64,67] with a focus on generating quality pseudo-labels using temporal information [43,67] or an IoU scoring criterion for historical pseudo-labels [63,64]. In particular, while Yang et al [63,64] has drastically improved the performance over previous works, it is not practical for a lidar that can adjust its scan pattern in real-time.…”

“…A popular approach is the use of self-training [43,63,64,67] with a focus on generating quality pseudo-labels using temporal information [43,67] or an IoU scoring criterion for historical pseudo-labels [63,64]. In particular, while Yang et al [63,64] has drastically improved the performance over previous works, it is not practical for a lidar that can adjust its scan pattern in real-time. The method would need to be fine-tuned for every adjustment of the scan pattern, and in practice, this fine-tuned model would need to be constantly swapped according to the adjusted scan.…”