CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

Sanghi, Aditya; Chu, Hang; Lambourne, Joseph G.; Wang, Ye; Cheng, Chin-Yi; Fumero, Marco

doi:10.1109/cvpr52688.2022.01805

Cited by 138 publications

(38 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The fully-supervised method [6,8,25] uses ground truth text and the paired 3D objects with explicit 3D representations as training data. Specifically, CLIP-Forge [43] uses a two-stage training scheme, which consists of shape autoencoder training, and conditional normalizing flow training. VQ-VAE [44] performs zero-shot training with 3D voxel data by utilizing the pretrained CLIP model [37].…”

Section: Text-guided 3d Shape Generationmentioning

confidence: 99%

“…To demonstrate our 3D generation quality, we compare existing works [25,43,50] from three aspects: 1) rendered 2D image quality using FID, 2) text-image relevance using R-Precision, and 3) 3D geometry quality using FPD. In Tab.…”

Section: Quantitative Comparisonmentioning

confidence: 99%

“…In order to control the generated 3D objects from text, prior text-to-3D generation works [43,44] leverage the pretrained vision-language alignment model CLIP [37], such that they can only use 3D shape data to achieve zero-shot learning. For example, Dream Fields [13] combines the advantages of CLIP and NeRF [27], which can produce both 3D representations and renderings.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision

Wei¹,

Wang²,

Feng³

et al. 2023

Preprint

View full text Add to dashboard Cite

a red hatchback car" "a gray armless chair" "a wooden office table" "a black motorbike" Figure 1. Our proposed model is capable of generating detailed textured shapes based on given text prompts. We demonstrate the effectiveness of our approach by showcasing generated results for four distinct object classes: Car, Chair, Table, and Motorbike. Both the textured meshes and geometries are presented, with visualizations rendered using ChimeraX [10].

show abstract

Section: Text-guided 3d Shape Generationmentioning

confidence: 99%

Section: Quantitative Comparisonmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision

Wei¹,

Wang²,

Feng³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…It encodes text and 3D shapes separately into the same latent space. However, large-scale 3D-text datasets are still difficult to obtain, so ClipForge [22] bypasses this problem with the aid of the CLIP model on text-image matching. CLIP-Mesh [23] also uses the CLIP model to measure the matching degree between the image rendered by the grid model and the text, so as to optimize the entire model parameters.…”

Section: Introductionmentioning

confidence: 99%

Zero3D: Semantic-Driven Multi-Category 3D Shape Generation

Han¹,

Liu²,

Shen³

2023

Preprint

View full text Add to dashboard Cite

Semantic-driven 3D shape generation aims to generate 3D objects conditioned on text. Previous works face problems with single-category generation, low-frequency 3D details, and requiring a large number of paired datasets for training. To tackle these challenges, we propose a multi-category conditional diffusion model. Specifically, 1) to alleviate the problem of lack of large-scale paired data, we bridge the text, 2D image and 3D shape based on the pre-trained CLIP model, and 2) to obtain the multi-category 3D shape feature, we apply the conditional flow model to generate 3D shape vector conditioned on CLIP embedding. 3) to generate multicategory 3D shape, we employ the hidden-layer diffusion model conditioned on the multi-category shape vector, which greatly reduces the training time and memory consumption.

show abstract

“…An increasingly growing area of research is the creation of 3D models of objects based on text descriptions. Depending on the approach, these objects are created as point cloud (Yang et al, 2019;Achlioptas et al, 2018), voxel (Sanghi et al, 2022;Chen et al, 2018), mesh (Nash et al, 2020) or implicite representation Mescheder et al, 2019). Or alternatively get the most suitable object from an object database (Textto-shape retrieval; Ruan et al, 2022).…”

Section: Text-to-shape Generationmentioning

confidence: 99%

Toward context-based text-to-3D scene generation

Henlein¹

View full text Add to dashboard Cite

People can describe spatial scenes with language and, vice versa, create images based on linguistic descriptions. However, current systems do not even come close to matching the complexity of humans when it comes to reconstructing a scene from a given text. Even the ever-advancing development of better and better Transformer-based models has not been able to achieve this so far. This task, the automatic generation of a 3D scene based on an input text, is called text-to-3D scene generation. The key challenge, and focus of this dissertation, now relate to the following topics: (a) Analyses of how well current language models understand spatial information, how static embeddings compare, and whether they can be improved by anaphora resolution. (b) Automated resource generation for context expansion and grounding that can help in the creation of realistic scenes. (c) Creation of a VR-based text-to-3D scene system that can be used as an annotation and active-learning environment, but can also be easily extended in a modular way with additional features to solve more contexts in the future. (d) Analyze existing practices and tools for digital and virtual teaching, learning, and collaboration, as well as the conditions and strategies in the context of VR. In the first part of this work, we could show that static word embeddings do not benefit significantly from pronoun substitution. We explain this result by the loss of contextual information, the reduction in the relative occurrence of rare words, and the absence of pronouns to be substituted. But we were able to we have shown that both static and contextualizing language models appear to encode object knowledge, but require a sophisticated apparatus to retrieve it. The models themselves in combination with the measures differ greatly in terms of the amount of knowledge they allow to extract. Classifier-based variants perform significantly better than the unsupervised methods from bias research, but this is also due to overfitting. The resources generated for this evaluation are later also an important component of point three. In the second part, we present AffordanceUPT, a modularization of UPT trained on the HICO-DET dataset, which we have extended with Gibsonien/telic annotations. We then show that AffordanceUPT can effectively make the Gibsonian/telic distinction and that the model learns other correlations in the data to make such distinctions (e.g., the presence of hands in the image) that have important implications for grounding images to language. The third part first presents a VR project to support spatial annotation respectively IsoSpace. The direct spatial visualization and the immediate interaction with the 3D objects should make the labeling more intuitive and thus easier. The project will later be incorporated as part of the Semantic Scene Builder (SeSB). The project itself in turn relies on the Text2SceneVR presented here for generating spatial hypertext, which in turn is based on the VAnnotatoR. Finally, we introduce Semantic Scene Builder (SeSB), a VR-based text-to-3D scene framework using Semantic Annotation Framework (SemAF) as a scheme for annotating semantic relations. It integrates a wide range of tools and resources by utilizing SemAF and UIMA as a unified data structure to generate 3D scenes from textual descriptions and also supports annotations. When evaluating SeSB against another state-of-the-art tool, it was found that our approach not only performed better, but also allowed us to model a wider variety of scenes. The final part reviews existing practices and tools for digital and virtual teaching, learning, and collaboration, as well as the conditions and strategies needed to make the most of technological opportunities in the future.

show abstract

CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

Cited by 138 publications

References 58 publications

TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision

TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision

Zero3D: Semantic-Driven Multi-Category 3D Shape Generation

Toward context-based text-to-3D scene generation

Contact Info

Product

Resources

About