Voxel-MAE: Masked Autoencoders for Pre-training Large-scale Point Clouds

Chen, Min; Xu, Xiang; Zhao, Debin; Liang, Xiao; Nie, Yiming; Dai, Bin

doi:10.48550/arxiv.2206.09900

Cited by 5 publications

(5 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the 3D domain, methods like [28], [33] utilize local reconstruction techniques for pretraining point cloud models. The pretraining transformer networks provide a robust initialization, which is beneficial for downstream tasks.…”

Section: B Backbone Pretraining and Scalingmentioning

confidence: 99%

PillarNeSt: Embracing Backbone Scaling and Pretraining for Pillar-based 3D Object Detection

Mao,

Wang,

Zhang

et al. 2024

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

Pillar-based 3D object detectors mainly employ randomly initialized 2D convolution neural network (ConvNet) for feature extraction and fail to enjoy the benefits from the backbone scaling and pretraining in the image domain. This paper shows the effectiveness of 2D backbone scaling and pretraining for pillar-based 3D object detectors. For better backbone scaling, we first introduce several design principles for point cloud backbone, to tackle the sparsity of point cloud and improve the effective receptive field. The backbone scaling is achieved by adaptively designed based on the model size. For backbone pretraining, we propose a weight adaptation module, to transfer the image knowledge obtained by pretraining on large-scale image datasets for the point cloud. Our proposed pillar-based detector, termed PillarNeSt, outperforms the existing 3D object detectors by a large margin on the nuScenes and Argoversev2 datasets. Code is released at https://github.com/WayneMao/PillarNeSt.

show abstract

Section: B Backbone Pretraining and Scalingmentioning

confidence: 99%

PillarNeSt: Embracing Backbone Scaling and Pretraining for Pillar-based 3D Object Detection

Mao,

Wang,

Zhang

et al. 2024

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

show abstract

“…In terms of specific task improvements, Point-M2AE [127] (2022.5) improves the learning of irregular point clouds 3D representation by modifying the encoder and decoder VOLUME 11, 2023 into a pyramid architecture, capturing fine-grained and highlevel semantic information of 3D shapes. VoxelMAE [128] (2022.6) addresses the sparse density of points and large variations in the same scene in autonomous driving point clouds. It specifically designs the discrimination between empty and non-empty points, similar to [126].…”

Section: ) Point Cloudsmentioning

confidence: 99%

Masked Autoencoders in Computer Vision: A Comprehensive Survey

Zhou,

Liu

2023

IEEE Access

View full text Add to dashboard Cite

Masked autoencoders (MAE) is a deep learning method based on Transformer. Originally used for images, it has now been extended to video, audio, and some other temporal prediction tasks. In the field of computer vision, MAE performs well in classification, prediction, and target detection tasks. In terms of specific application, MAE has made many achievements in medical treatment, geography, 3D point cloud and machine troubleshooting. Since its introduction at the end of 2021, there have been more than 300 related preprints, and MAE has been significantly performed in tier one computer vision conferences during 2022 and 2023. In view of the current popularity of MAE and its future development prospects, we conduct a relatively comprehensive survey of MAE mainly covering officially published articles so far. We comb through and classify the improvements in MAE, demonstrating relatively representative applications in computer vision. Finally, as a summary, we discuss the possible future research directions and development areas based on the characteristics of MAE, hoping our work could be a reference for the future work of MAE.INDEX TERMS Computer vision survey, MAE, masked autoencoders, masked image modeling.

show abstract

“…Nevertheless, our proposed system boasts the flexibility to utilize disparate tasks and learning frameworks. With the current advancements in point-cloud pretraining [135], [136] and auto-labeling technologies [24], [137], we now have more choices for practical task learning that are independent of manual labels. In our framework, model training at the local end can be easily replaced with these models and methodologies.…”

Section: B Limitationsmentioning

confidence: 99%

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Wang

Chaokang

Shen

et al. 2021

Preprint

View full text Add to dashboard Cite

3D scene flow presents the 3D motion of each point in the 3D space, which forms the fundamental 3D motion perception for autonomous driving and server robots. Although the RGBD camera or LiDAR capture discrete 3D points in space, the objects and motions usually are continuous in the macro world. That is, the objects keep themselves consistent as they flow from the current frame to the next frame. Based on this insight, the Generative Adversarial Networks (GAN) is utilized to self-learn 3D scene flow with no need for ground truth. The fake point cloud of the second frame is synthesized from the predicted scene flow and the point cloud of the first frame. The adversarial training of the generator and discriminator is realized through synthesizing indistinguishable fake point cloud and discriminating the real point cloud and the synthesized fake point cloud. The experiments on KITTI scene flow dataset show that our method realizes promising results without ground truth. Just like a human observing a real-world scene, the proposed approach is capable of determining the consistency of the scene at different moments in spite of the exact flow value of each point is unknown in advance. Corresponding author(s) Email: wanghesheng@sjtu.edu.cn

show abstract

Voxel-MAE: Masked Autoencoders for Pre-training Large-scale Point Clouds

Cited by 5 publications

References 32 publications

PillarNeSt: Embracing Backbone Scaling and Pretraining for Pillar-based 3D Object Detection

PillarNeSt: Embracing Backbone Scaling and Pretraining for Pillar-based 3D Object Detection

Masked Autoencoders in Computer Vision: A Comprehensive Survey

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Contact Info

Product

Resources

About