ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation

Abstract

Text-guided 3D shape generation remains challenging due to the absence of large paired text-shape dataset, the substantial semantic gap between these two modalities, and the structural complexity of 3D shapes. This paper presents a new framework called Image as Stepping Stone (ISS) for the task by introducing 2D image as a stepping stone to connect the two modalities and to eliminate the need for paired text-shape data. Our key contribution is a two-stage feature-space-alignment approach that maps CLIP features to shapes by harnessing a pre-trained single-view reconstruction (SVR) model with multi-view supervisions: first map the CLIP image feature to the detail-rich shape space in the SVR model, then map the CLIP text feature to the shape space and optimize the mapping by encouraging CLIP consistency between the input text and the rendered images. Further, we formulate a text-guided shape stylization module to dress up the output shapes with novel structures and textures. Beyond existing works on 3D shape generation from text, our new approach is general for creating shapes in a broad range of categories, without requiring paired text-shape data. Experimental results manifest that our approach outperforms the state-of-the-arts and our baselines in terms of fidelity and consistency with text. Further, our approach can stylize the generated shapes with both realistic and fantasy structures and textures.A green bedside lamp.

Evaluation Metrics for Intelligent Generation of Graphical Game Assets: A Systematic Survey-Based Framework

“…FID utilises Frechet distance between the two image distributions, and assumes that the two are Gaussian. This is applied to 2D assets [7], [64], [105], [106], [107], [108], [109], 3D assets via 3D classifiers [37], [110] or rasterisation to 2D form [111], [112].The Frechet point cloud distance extends FID for applications in assessing the similarity of pointbased 3D shapes [113]. This has been used to evaluate many deep-learning based 3D point-cloud [71], [113], [114] and mesh [67] generators.…”

X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance

“…Some notable examples of such methods include CLIP-NeRF [58], PureCLIPNeRF [24], and Dream-Fields [19]. Additionally, recent studies have explored the fusion of CLIP with other algorithms, such as ISS [27] with SVR [37], CLIP-Forge [50] using a normalizing flow network [10], and AvatarCLIP [15] leveraging SMLP [30]. Furthermore, the diffusion model [48] has recently demonstrated impressive results in text-to-image generation, leading to its integration into the text-to-3D generation process.…”