Semantics‐Guided Latent Space Exploration for Shape Generation

Jahan, Tansin; Guan, Yanran; Kaick, Oliver van

doi:10.1111/cgf.142619

Cited by 12 publications

(6 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AI-generated content (AIGC), also known as generative AI (GAI), recently gained a surge of development (Zhang et al 2023a,b;Cao et al 2023;Balaji et al 2022), covering several areas such as image (Ramesh et al 2021Saharia et al 2022;Yu et al 2022;Rombach et al 2022), text (Raffel et al 2020;Radford et al 2018Radford et al , 2019Brown et al 2020;Ope-nAI 2023;Vinyals et al 2015), 3D (Fu et al 2022;Jahan, Guan, and Van Kaick 2021;Liu et al 2022;Mildenhall et al 2021), and speech (Qian et al 2014;Ze, Senior, and Schuster 2013;Zen and Sak 2015). Since ChatGPT emerged, people have been amazed by its performance while recognizing the reasoning potential in the generated text content (Bang et al 2023).…”

Section: Introductionmentioning

confidence: 99%

Image Content Generation with Causal Reasoning

Li,

Fan,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

The emergence of ChatGPT has once again sparked research in generative artificial intelligence (GAI). While people have been amazed by the generated results, they have also noticed the reasoning potential reflected in the generated textual content. However, this current ability for causal reasoning is primarily limited to the domain of language generation, such as in models like GPT-3. In visual modality, there is currently no equivalent research. Considering causal reasoning in visual content generation is significant. This is because visual information contains infinite granularity. Particularly, images can provide more intuitive and specific demonstrations for certain reasoning tasks, especially when compared to coarse-grained text. Hence, we propose a new image generation task called visual question answering with image (VQAI) and establish a dataset of the same name based on the classic Tom and Jerry animated series. Additionally, we develop a new paradigm for image generation to tackle the challenges of this task. Finally, we perform extensive experiments and analyses, including visualizations of the generated content and discussions on the potentials and limitations. The code and data are publicly available under the license of CC BY-NC-SA 4.0 for academic and non-commercial usage at: https://github.com/IEIT-AGI/MIX-Shannon/blob/main/projects/VQAI/lgd_vqai.md.

show abstract

Section: Introductionmentioning

confidence: 99%

Image Content Generation with Causal Reasoning

Li,

Fan,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Existing works (Chen et al (2018); Jahan et al (2021); Liu et al (2022)) typically rely on paired text-shape data for model training. Yet, collecting 3D shapes is already very challenging on its own, let alone the tedious manual annotations needed to construct the text-shape pairs.…”

Section: Introductionmentioning

confidence: 99%

ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation

Liu¹,

Dai²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Text-guided 3D shape generation remains challenging due to the absence of large paired text-shape dataset, the substantial semantic gap between these two modalities, and the structural complexity of 3D shapes. This paper presents a new framework called Image as Stepping Stone (ISS) for the task by introducing 2D image as a stepping stone to connect the two modalities and to eliminate the need for paired text-shape data. Our key contribution is a two-stage feature-space-alignment approach that maps CLIP features to shapes by harnessing a pre-trained single-view reconstruction (SVR) model with multi-view supervisions: first map the CLIP image feature to the detail-rich shape space in the SVR model, then map the CLIP text feature to the shape space and optimize the mapping by encouraging CLIP consistency between the input text and the rendered images. Further, we formulate a text-guided shape stylization module to dress up the output shapes with novel structures and textures. Beyond existing works on 3D shape generation from text, our new approach is general for creating shapes in a broad range of categories, without requiring paired text-shape data. Experimental results manifest that our approach outperforms the state-of-the-arts and our baselines in terms of fidelity and consistency with text. Further, our approach can stylize the generated shapes with both realistic and fantasy structures and textures.A green bedside lamp.

show abstract

“…Finally, an obvious benefit of our method in comparison to methods based on natural language [13,24] is that it needs no manual labeling of the data to enable the exploration. In comparison to sketching approaches [18], artistic skills are not required from the user, while in comparison to classic parametric models [8], the requirements on data pre-processing are not as stringent.…”

Section: Overview Of Our Methodsmentioning

confidence: 99%

“…Learning of the latent space can be accomplished with the use of an autoencoder network composed of an encoder and decoder [24]. The encoder takes a 3D shape as input, e.g., encoded as a 3D volume, and returns the latent representation of the shape.…”

Section: Introductionmentioning

confidence: 99%

“…Finally, a third category of approaches enables the exploration of latent spaces with the use of natural language. For example, Jahan et al [24] introduce a method to enable the exploration of latent spaces through keywords, while Chen et al [13] introduce a method to directly synthesize a shape from a description in natural language. These methods require additional pre-processing for associating keywords to shapes or establishing a connection between shapes and sentences in natural language.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exploration of Latent Spaces of 3D Shapes via Isomap and Inverse Mapping

Rowaida¹

View full text Add to dashboard Cite

In this thesis, we introduce a method for exploring a latent space of 3D shapes learned by a deep neural network. The main idea of our method is to enable the exploration of the latent space through the navigation of a 2D embedding. The method is based on a combination of Isomap dimensionality reduction and an inverse mapping function. Specifically, given a dataset of 3D shapes, we first train an autoencoder neural network to learn a latent representation for the dataset. Then, we reduce the dimensionality of the latent space to two dimensions with the Isomap method. The 2D embedding of the latent space allows a user to easily navigate through the embedding and sample new points. Our method then translates the sampled points back into latent vectors with an inverse mapping function, while the latent vectors are decoded by the neural network into 3D shapes that can be inspected by the user. The inverse mapping is made possible by posing it as a radial basis function (RBF) scattered interpolation problem. We demonstrate with qualitative experiments that our exploration method has advantages compared to alternative approaches such as interpolation and principal component analysis. We also show with quantitative experiments that our method enables a meaningful exploration of the latent space.iii Kaick, my research supervisor, for giving me the opportunity and immense support to make this research a reality. His encouragement, guidance and patience truly helped me to shape this research into what it is today. I would also like to thank Dr. David Mould, Dr. WonSook Lee for their thorough review of my work. Their valuable suggestions and useful critiques towards the thesis improved the quality of the work. I would like to thank GIGL lab and all the members for their encouragement and suggestions to prepare myself for the defense. I am also grateful to the School of Computer Science, Carleton University for accepting me as a research student and facilitating me with financial assistance to complete my degree. At last, I would like to thank my parents, sister, family members and my beloved husband for their unconditional love and support that encouraged me to complete my research.

show abstract

Semantics‐Guided Latent Space Exploration for Shape Generation

Cited by 12 publications

References 33 publications

Image Content Generation with Causal Reasoning

Image Content Generation with Causal Reasoning

ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation

Exploration of Latent Spaces of 3D Shapes via Isomap and Inverse Mapping

Contact Info

Product

Resources

About