Controllable high-fidelity facial performance transfer

Xu, Feng; Chai, Jinxiang; Liu, Yilong; Tong, Xin

doi:10.1145/2601097.2601210

Cited by 33 publications

(15 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thies et al [2] developed a system that transfers expression changes from the source to the target actor based on [15] and achieves real-time performance. Xu et al [19] designed a facial expression transfer and editing technique for high-fidelity facial performance data. Moreover, other flow-based approaches [3,4] are also proposed to transfer facial expression to different face meshes.…”

Section: Deformation Transfermentioning

confidence: 99%

Cartoonish sketch-based face editing in videos using identity deformation transfer

Zhao

Han

Peng

et al. 2019

Computers & Graphics

View full text Add to dashboard Cite

A B S T R A C TWe address the problem of using hand-drawn sketches to create exaggerated deformations to faces in videos, such as enlarging the shape or modifying the position of eyes or mouth. This task is formulated as a 3D face model reconstruction and deformation problem. We first recover the facial identity and expressions from the video by fitting a face morphable model for each frame. At the same time, user's editing intention is recognized from input sketches as a set of facial modifications. Then a novel identity deformation algorithm is proposed to transfer these facial deformations from 2D space to the 3D facial identity directly while preserving the facial expressions. After an optional stage for further refining the 3D face model, these changes are propagated to the whole video with the modified identity. Both the user study and experimental results demonstrate that our sketching framework can help users effectively edit facial identities in videos, while high consistency and fidelity are ensured at the same time.

show abstract

Section: Deformation Transfermentioning

confidence: 99%

Cartoonish sketch-based face editing in videos using identity deformation transfer

Zhao

Han

Peng

et al. 2019

Computers & Graphics

View full text Add to dashboard Cite

show abstract

“…Similar to previous sketch‐based interfaces [ZNA07, LCXS09, MAO*11], our 2D editing tool allows for intuitive control of lip shapes and is easy to use even for non‐expert users. This idea for simple facial animation editing is akin to keyframe‐based performance editing tools, such as those used during expression retargeting [SLS*12, XCLT14].…”

Section: Related Workmentioning

confidence: 99%

User‐Guided Lip Correction for Facial Performance Capture

Dinev

Beeler

Bradley

et al. 2018

Computer Graphics Forum

View full text Add to dashboard Cite

Facial performance capture is the primary method for generating facial animation in video games, feature films and virtual environments, and recent advances have produced very compelling results. Still, one of the most challenging regions is the mouth, which often contains systematic errors due to the complex appearance and occlusion/dis‐occlusion of the lips. We present a novel user‐guided approach to correcting these common lip shape errors present in traditional capture systems. Our approach is to allow a user to manually correct a small number of problematic frames, and then our system learns the types of corrections desired and automatically corrects the entire performance. As correcting even a single frame using traditional 3D sculpting tools can be time consuming and require great skill, we also propose a simple and fast 2D sketch‐based method for generating plausible lip corrections for the problematic key frames. We demonstrate our results on captured performances of three different subjects, and validate our method with an additional sequence that contains ground truth lip reconstructions.

show abstract

“…Moreover, we extend our approach by combing expression cloning technique, overcoming the limitation in [17] that only face model having the priori can be edited. Recently, spacetime editing methods [15,[18][19][20][21] that can propagate the editing on a single face frame across the whole sequence have been explored. For example, [19] built a Poisson equation to propagate user's modifications at any frames to the entire sequence.…”

Section: Related Workmentioning

confidence: 99%

Interactive facial expression editing based on spatio-temporal coherency

2017

View full text Add to dashboard Cite

We present a novel approach for interactively and intuitively editing 3D facial animation in this paper. It determines a new expression by combining the user-specified constraints with the priors contained in a pre-recorded facial expression set, which effectively overcomes the generation of an unnatural expression caused by only user-constraints. The approach is based on the framework of example-based linear interpolation. It adaptively segments the face model into soft regions based on user-interaction. In dependently modeling each region, we propose a new function to estimate the blending weight of each example that matches the user-constraints as well as the spatio-temporal properties of the face set. In blending the regions into a single expression, we present a new criterion that fully exploits the spatial proximity and the spatio-temporal motion consistency over the face set to measure the coherency between vertices and use the coherency to reasonably propagate the influence of each region to the entire face model. Experiments show that our approach, even with inappropriate user's edits, can create a natural expression that optimally satisfies the user-desired goal.

show abstract

Controllable high-fidelity facial performance transfer

Cited by 33 publications

References 40 publications

Cartoonish sketch-based face editing in videos using identity deformation transfer

Cartoonish sketch-based face editing in videos using identity deformation transfer

User‐Guided Lip Correction for Facial Performance Capture

Interactive facial expression editing based on spatio-temporal coherency

Contact Info

Product

Resources

About