Expression Packing: As‐Few‐As‐Possible Training Expressions for Blendshape Transfer

Abstract

To simplify and accelerate the creation of blendshape rigs, using a template rig is a common procedure, especially during the creation of digital doubles. Blendshape transfer methods facilitate copy and paste functionality of the blendshapes from the template model to the digital double. However, for adequate personalization, such methods require a set of scanned training expressions of the original actor. So far, the semantics of the facial expressions to scan have been defined manually. In contrast, we formulate the semantics of the facial expressions as an integer optimization of the blendshape weights. By combining different blendshapes of the template model, our method creates facial expressions that serve as semantic references during scanning. Our method guarantees to compute as‐few‐as‐possible training expressions with minimal overlap of activated blendshapes. If the number of training expressions is limited, blendshapes are selected based on their power to personalize the resulting blendshapes compared to generic blendshape transfer methods.

Investigating perceptually based models to predict importance of facial blendshapes

“…Additionally, we found that some facial action units were more perceptually noticeable than others, and provide a table showing the order of importance (Table 2). This perceptual ordering will be useful for game developers for tasks that require an order of blendshapes, such as level-of-detail blendshape reduction methods [Costigan et al 2016], or example creation for blendshape transfer [Carrigan et al 2020]. By identifying and removing blendshapes of lower visual saliency, which equates to simply removing rows from the blendshape matrix, we can save both memory and computation required.…”

“…games and other real-time applications, with the aim of reducing the number of blendshapes needed for animating a rig [Costigan et al 2016], or prioritising which blendshapes to include in expressions for example-based rig creation algorithms [Carrigan et al 2020]. Additionally, algorithms that create or alter facial geometry are usually evaluated against ground-truth facial meshes using standard geometry error metrics, however, we postulate that standard error-metrics may not be sufficient to determine how perceptually different the results are to the ground-truth.…”

Dynamic facial asset and rig generation from a single scan

“…To build robust face rigs, we need to reconstruct a dynamic expression model that faithfully captures the subject's specific facial movements. A full set of personalized blendshapes for a specific subject can be built from 3D scan data of the same subject [Carrigan et al 2020;Huang et al 2011;Li et al 2010;Weise et al 2009;Zhang et al 2004]. These methods can reconstruct expressions that capture the target's personal expressions, but a large set of action units or sparse expressions are required as input.…”

Generating High-Resolution 3D Faces and Bodies Using VQ-VAE-2 with PixelSNAIL Networks on 2D Representations

“…There are many reasons behind this, from the growth in computational power to the economic benefit of using a parametric model to simulate physical phenomena. Today, 3D models serve many fields, including animation of characters [ 1 ] and faces [ 2 , 3 , 4 ], recognition of expressions [ 5 ], face recognition [ 6 ], and inferring body shapes and measurement to be used, for example, in the clothing industry, for virtual try-on [ 7 ], or in the medical field to estimate fat distribution. However, the applications are often limited due to privacy and sensitive information constraints that reduce or block data sharing and aggregation from multiple sources.…”