CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Khalid, Nasir Mohammad; Xie, Tianhao; Belilovsky, Eugene; Popa, Tiberiu

doi:10.1145/3550469.3555392

Cited by 114 publications

(31 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More related to our work, CLIP-Mesh [20] and Dream Fields [16] do so by using the CLIP embedding and can condition 3D generation on text. Our model is built on the recent Dream Fusion approach [33], which builds on a similar idea using a diffusion model as prior.…”

Section: Related Workmentioning

confidence: 99%

RealFusion: 360° Reconstruction of Any Object from a Single Image

Melas-Kyriazi¹,

Rupprecht²,

Laina³

et al. 2023

Preprint

View full text Add to dashboard Cite

https://lukemelas.github.io/realfusion Figure 1. RealFusion generates a full 360 • reconstruction of any object given a single image of it (left column). It does so by leveraging an existing diffusion-based 2D image generator. From the given image, it synthesizes a prompt that causes the diffusion model to "dream up" other views of the object. It then extracts a neural radiance field from the original image and the diffusion model-based prior, thereby reconstructing the object in full. Both appearance and geometry are reconstructed faithfully and extrapolated in a plausible manner (see the textured and shaded reconstructions from different viewpoints).

show abstract

Section: Related Workmentioning

confidence: 99%

RealFusion: 360° Reconstruction of Any Object from a Single Image

Melas-Kyriazi¹,

Rupprecht²,

Laina³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…3D Generation Guided by Text-Image Models: Another line of work leverages image-text models trained on massive datasets to provide supervision for 3D generation. For instance, methods like DreamField [13], CLIP-Mesh [14] and PureCLIPNeRF [17] learn to generate 3D models by optimizing the CLIP similarity scores between rendered images and the input text prompts. Instead of relying on CLIP which might be insufficient to capture high frequency details, more recent methods [26,18,34] use text-to-image generation models to provide alignment supervision between text prompts and rendered images, and generate more photo-realistic 3D models.…”

Section: Related Workmentioning

confidence: 99%

3DGen: Triplane Latent Diffusion for Textured Mesh Generation

Gupta¹,

Xiong²,

Nie³

et al. 2023

Preprint

View full text Add to dashboard Cite

Latent diffusion models for image generation have crossed a quality threshold which enabled them to achieve mass adoption. Recently, a series of works have made advancements towards replicating this success in the 3D domain, introducing techniques such as point cloud VAE, triplane representation, neural implicit surfaces and differentiable rendering based training. We take another step along this direction, combining these developments in a two-step pipeline consisting of 1) a triplane VAE which can learn latent representations of textured meshes and 2) a conditional diffusion model which generates the triplane features. For the first time this architecture allows conditional and unconditional generation of high quality textured or untextured 3D meshes across multiple diverse categories in a few seconds on a single GPU. It outperforms previous work substantially on image-conditioned and unconditional generation on mesh quality as well as texture generation. Furthermore, we demonstrate the scalability of our model to large datasets for increased quality and diversity. 1 * Equal contributions 1 We will release our code and trained models.

show abstract

“…Neural fields are an intriguing way to fully generate 3D models because, unlike meshes, they don't depend on topological properties such as genus and subdivision [28,15]. Initial generative text-to-3D attempts with NeRF backbones took advantage of a robust language model [53] to align each rendered view on some textual condition [21,73].…”

Section: Related Workmentioning

confidence: 99%