PointCLIP: Point Cloud Understanding by CLIP

Zhang, Renrui; Guo, Ziyu; Zhang, Wei; Li, Kunchang; Miao, Xupeng; Cui, Bin; Qiao, Yu; Gao, Peng; Li, Hongsheng

doi:10.1109/cvpr52688.2022.00836

Cited by 212 publications

(67 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results are shown in Table 3. Following PointCLIP (Zhang et al, 2022c), We use prompt templates with the category label to generate text features. From Table 3, it can be seen that our RECON surpasses all the zero-shot methods with CNN-based or Transformer-based backbones.…”

Section: Methodsmentioning

confidence: 99%

“…Cross-Point (Afham et al, 2022) uses both inter and intra-modal contrastive learning. PointCLIP (Zhang et al, 2022c) realizes image-point alignment by projecting point clouds to 2D depth images. Different from these methods, our work focuses on cross-modal contrastive learning by global feature alignment like Radford et al (2021), which is guided by generative modeling (Pang et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

Qi¹,

Dong²,

Fan³

et al. 2023

Preprint

View full text Add to dashboard Cite

Mainstream 3D representation learning approaches are built upon contrastive or generative modeling pretext tasks, where great improvements in performance on various downstream tasks have been achieved. However, by investigating the methods of these two paradigms, we find that (i) contrastive models are data-hungry that suffer from a representation over-fitting issue; (ii) generative models have a data filling issue that shows inferior data scaling capacity compared to contrastive models. This motivates us to learn 3D representations by sharing the merits of both paradigms, which is non-trivial due to the pattern difference between the two paradigms. In this paper, we propose contrast with reconstruct (RECON) that unifies these two paradigms. RECON is trained to learn from both generative modeling teachers and cross-modal contrastive teachers through ensemble distillation, where the generative student guides the contrastive student. An encoder-decoder style RECON-block is proposed that transfers knowledge through cross attention with stop-gradient, which avoids pretraining over-fitting and pattern difference issues. RECON achieves a new state-of-the-art in 3D representation learning, e.g., 91.26% accuracy on ScanObjectNN. Codes will be released at https://github.com/qizekun/ReCon.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

Qi¹,

Dong²,

Fan³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…With a high-capacity text encoder and visual encoder, CLIP (Radford et al, 2021) aligns of text and visual elements in heterogeneous embedding space. Several studies extend the applicability of CLIP to other tasks such as object detection (Gu et al, 2021), semantic segmentation (Wang et al, 2022;Li et al, 2022;Zhang et al, 2022) and image editing (Patashnik et al, 2021;Gal et al, 2021). Among them, PointClip (Zhang et al, 2022) is most related to our work.…”

Section: Related Workmentioning

confidence: 95%

“…Several studies extend the applicability of CLIP to other tasks such as object detection (Gu et al, 2021), semantic segmentation (Wang et al, 2022;Li et al, 2022;Zhang et al, 2022) and image editing (Patashnik et al, 2021;Gal et al, 2021). Among them, PointClip (Zhang et al, 2022) is most related to our work. It learns transferable visual concepts by leveraging CLIP's pre-trained knowledge for zero-shot point cloud recognition.…”

Section: Related Workmentioning

confidence: 95%

Label Name is Mantra: Unifying Point Cloud Segmentation across Heterogeneous Datasets

Liang¹,

He²,

Xiao³

et al. 2023

Preprint

View full text Add to dashboard Cite

Point cloud segmentation is a fundamental task in 3D vision that serves a wide range of applications. Although great progresses have been made these years, its practical usability is still limited by the availability of training data. Existing approaches cannot make full use of multiple datasets on hand due to the label mismatch among different datasets. In this paper, we propose a principled approach that supports learning from heterogeneous datasets with different label sets. Our idea is to utilize a pre-trained language model to embed discrete labels to a continuous latent space with the help of their label names. This unifies all labels of different datasets, so that joint training is doable. Meanwhile, classifying points in the continuous 3D space by their vocabulary tokens significantly increase the generalization ability of the model in comparison with existing approaches that have fixed decoder architecture. Besides, we also integrate prompt learning in our framework to alleviate data shifts among different data sources. Extensive experiments demonstrate that our model outperforms the state-of-the-art by a large margin.

show abstract

“…The seminal work CLIP [66] learns a joint language-vision embedding using more than 400 billion text-image pairs. The learned representation is semantically meaningful and expressive, thus has been adapted to various downstream tasks [82,71,49,79,65]. In this work, we adopt CLIP in the retrieval process.…”

Section: Related Workmentioning

confidence: 99%