ClipFace: Text-guided Editing of Textured 3D Morphable Models

Aneja, Shivangi; Thies, Justus; Dai, Angela; Nießner, Matthias

doi:10.48550/arxiv.2212.01406

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…enabling 3DMMlike control [BV99]) and thus do not offer the same level of interactivity. Lastly, while ClipFace [ATDN22] provides editability on 3D morphable models (3DMM) and their texture, it does not handle other facial features like hair.…”

Section: Categorymentioning

confidence: 99%

PVP: Personalized Video Prior for Editable Dynamic Portraits using StyleGAN

Lin,

Trevithick,

Cheng

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Portrait synthesis creates realistic digital avatars which enable users to interact with others in a compelling way. Recent advances in StyleGAN and its extensions have shown promising results in synthesizing photorealistic and accurate reconstruction of human faces. However, previous methods often focus on frontal face synthesis and most methods are not able to handle large head rotations due to the training data distribution of StyleGAN. In this work, our goal is to take as input a monocular video of a face, and create an editable dynamic portrait able to handle extreme head poses. The user can create novel viewpoints, edit the appearance, and animate the face. Our method utilizes pivotal tuning inversion (PTI) to learn a personalized video prior from a monocular video sequence. Then we can input pose and expression coefficients to MLPs and manipulate the latent vectors to synthesize different viewpoints and expressions of the subject. We also propose novel loss functions to further disentangle pose and expression in the latent space. Our algorithm shows much better performance over previous approaches on monocular video datasets, and it is also capable of running in real‐time at 54 FPS on an RTX 3080.

show abstract

Section: Categorymentioning

confidence: 99%

PVP: Personalized Video Prior for Editable Dynamic Portraits using StyleGAN

Lin,

Trevithick,

Cheng

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…Several approaches have used the expressive power of CLIP to relate text to 3D shape. This includes text-driven generation of stylised meshes (Michel et al, 2022); general text to shape generation (Sanghi et al, 2022); text-based texture and expression editing (Aneja et al, 2022); and full body 3D avatar creation and animation (Hong et al, 2022). However, none of these methods consider the generation of a fully parameterised model of the human face, including identity, such as a 3DMM.…”

Section: Introductionmentioning

confidence: 99%

Text2Face: A Multi-Modal 3D Face Model

Will¹,

Huber²,

Pears³

et al. 2023

Preprint

View full text Add to dashboard Cite

We present the first 3D morphable modelling approach, whereby 3D face shape can be directly and completely defined using a textual prompt. Building on work in multi-modal learning, we extend the FLAME head model to a common imageand-text latent space. This allows for direct 3D Morphable Model (3DMM) parameter generation and therefore shape manipulation from textual descriptions. Our method, Text2Face, has many applications; for example: generating police photofits where the input is already in natural language. It further enables multimodal 3DMM image fitting to sketches and sculptures, as well as images.

show abstract

ClipFace: Text-guided Editing of Textured 3D Morphable Models

Cited by 2 publications

References 0 publications

PVP: Personalized Video Prior for Editable Dynamic Portraits using StyleGAN

PVP: Personalized Video Prior for Editable Dynamic Portraits using StyleGAN

Text2Face: A Multi-Modal 3D Face Model

Contact Info

Product

Resources

About