Video-based facial re-animation

Paier, Wolfgang; Kettern, Markus; Hilsmann, Anna; Eisert, Peter

doi:10.1145/2824840.2824843

Cited by 10 publications

(15 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…oral cavity) or simply ignore them. Other works like [Borshukov et al 2006;Carranza et al 2003;Casas et al 2014;Dale et al 2011;Fechteler et al 2014Fechteler et al , 2016Kilner et al 2006;Lipski et al 2011;Paier et al 2015Paier et al , 2017, try to circumvent these limitations by employing image-based rendering approaches, where the facial performance is captured in geometry and texture space. [Dale et al 2011] present a low cost system for facial performance transfer.…”

Section: Related Work 21 Modelling Of Facial Expressionsmentioning

confidence: 99%

“…Similar techniques were used in [Fechteler et al 2014], [Fechteler et al 2016] or [Thies et al 2015]. Another image based animation strategy was proposed in [Casas et al 2014;Paier et al 2015Paier et al , 2017, where low resolution 3D meshes were combined with highly detailed dynamic textures that allow generating high quality renderings. Based on previously captured video and geometry, they create a database of atomic geometry and texture sequences that can be re-arranged to synthesise novel performances.…”

Section: Related Work 21 Modelling Of Facial Expressionsmentioning

confidence: 99%

“…Together, geometry and texture represent almost the full appearance of the actress face in each frame. In order to generate high quality dynamic textures, we rely on a graph-cut based approach that implements a temporal consistency constraint [Paier et al 2015]. This method simultaneously optimises three data terms (5) to create a visually pleasing (i.e.…”

Section: Texture Extractionmentioning

confidence: 99%

See 2 more Smart Citations

Neural Face Models for Example-Based Visual Speech Synthesis

Paier

Hilsmann

Eisert

2020

European Conference on Visual Media Production

Self Cite

View full text Add to dashboard Cite

Section: Related Work 21 Modelling Of Facial Expressionsmentioning

confidence: 99%

Section: Related Work 21 Modelling Of Facial Expressionsmentioning

confidence: 99%

Section: Texture Extractionmentioning

confidence: 99%

See 1 more Smart Citation

Neural Face Models for Example-Based Visual Speech Synthesis

Paier

Hilsmann

Eisert

2020

European Conference on Visual Media Production

Self Cite

View full text Add to dashboard Cite

“…The captured videos can thereby be converted to texture space of the geometric proxy. Small sub‐sequences of the video footage can be re‐combined and mapped to the geometric proxy of a target video, which can be viewed from arbitrary directions [PKHE15].…”

Section: Related Workmentioning

confidence: 99%

Building a Large Database of Facial Movements for Deformation Model‐Based 3D Face Tracking

Sibbing

Kobbelt

2017

Computer Graphics Forum

View full text Add to dashboard Cite

We introduce a new markerless 3D face tracking approach for 2D videos captured by a single consumer grade camera. Our approach takes detected 2D facial features as input and matches them with projections of 3D features of a deformable model to determine its pose and shape. To make the tracking and reconstruction more robust we add a smoothness prior for pose and deformation changes of the faces. Our major contribution lies in the formulation of the deformation prior which we derive from a large database of facial animations showing different (dynamic) facial expressions of a fairly large number of subjects. We split these animation sequences into snippets of fixed length which we use to predict the facial motion based on previous frames. In order to keep the deformation model compact and independent from the individual physiognomy, we represent it by deformation gradients (instead of vertex positions) and apply a principal component analysis in deformation gradient space to extract the major modes of facial deformation. Since the facial deformation is optimized during tracking, it is particularly easy to apply them to other physiognomies and thereby re‐target the facial expressions. We demonstrate the effectiveness of our technique on a number of examples.

show abstract

“…This paper builds upon our previous works [8–14] and combines and extends them to build a full framework for the creation of animatable VV (AVV). The remainder of this paper is structured as follows.…”

Section: Introductionmentioning

confidence: 99%

Going beyond free viewpoint: creating animatable volumetric video of human performances

Hilsmann

Fechteler

Morgenstern

et al. 2020

IET Computer Vision

Self Cite

View full text Add to dashboard Cite

In this paper, we present an end-to-end pipeline for the creation of high-quality animatable volumetric video content of human performances. Going beyond the application of free-viewpoint volumetric video, we allow re-animation and alteration of an actor's performance through (i) the enrichment of the captured data with semantics and animation properties and (ii) applying hybrid geometry-and video-based animation methods that allow a direct animation of the high-quality data itself instead of creating an animatable model that resembles the captured data. Semantic enrichment and geometric animation ability are achieved by establishing temporal consistency in the 3D data, followed by an automatic rigging of each frame using a parametric shapeadaptive full human body model. Our hybrid geometry-and video-based animation approaches combine the flexibility of classical CG animation with the realism of real captured data. For pose editing, we exploit the captured data as much as possible and kinematically deform the captured frames to fit a desired pose. Further, we treat the face differently from the body in a hybrid geometry-and video-based animation approach where coarse movements and poses are modeled in the geometry only, while very fine and subtle details in the face, often lacking in purely geometric methods, are captured in video-based textures. These are processed to be interactively combined to form new facial expressions. On top of that, we learn the appearance of regions that are challenging to synthesize, such as the teeth or the eyes, and fill in missing regions realistically in an autoencoder-based approach. This paper covers the full pipeline from capturing and producing high-quality video content, over the enrichment with semantics and deformation properties for re-animation and processing of the data for the final hybrid animation.

show abstract

Video-based facial re-animation

Cited by 10 publications

References 34 publications

Neural Face Models for Example-Based Visual Speech Synthesis

Neural Face Models for Example-Based Visual Speech Synthesis

Building a Large Database of Facial Movements for Deformation Model‐Based 3D Face Tracking

Going beyond free viewpoint: creating animatable volumetric video of human performances

Contact Info

Product

Resources

About