Markerless reconstruction and synthesis of dynamic facial expressions

Building a Large Database of Facial Movements for Deformation Model‐Based 3D Face Tracking

“…On average a facial expression took 2 s to perform, which leads to a total number of about 96 000 frames to be reconstructed. To robustly and automatically perform this reconstruction and tracking task for this amount of frames, we used a system similar to the one suggested in [SHK11] and generated for each subject and each dynamic facial expression a sequence of meshes , where f represents the frame number within the respective dynamic facial expressions. To reconstruct the first frame, which shows the face in a neutral pose, we first identify facial features as seen by for synchronized cameras.…”

“…We rather use a mesh‐based tracking approach [SHK11, HDH*10]. Between two successive frames of a video sequence, this approach computes for every vertex of a two‐dimensional mesh a displacement vector such that an energy term is minimized, which is based on the formulation of the optical flow condition.…”

“…Since the position of eyes and mouth are known in this reconstruction, we can explicitly model the gaps between the eye lids and the mouth, and thereby deal with the potentially discontinuous movements at the borders of these gaps. For this work, we improved the additive formulation presented in [SHK11, HDH*10] and reformulate the mesh‐based tracking approach as an inverse compositional image alignment procedure based on the Lukas Kanade Tracker [BM04]. As we will see, this significantly shortens the computational time, which is important considering the amount of data we are dealing with.…”

“…It performs a GPU‐assisted stereo reconstruction for a set of camera pairs (for our four camera rig, we can define up to six of such pairs) followed by outlier filtering and disparity refinement and smoothing techniques. The main advantage of this technique is its performance: While the system used in [SHK09, SHK11] is only able to reconstruct 1600 points per second, this GPU‐assisted method reconstructs nearly 300 000 points per second. Most stereo reconstruction techniques heavily depend on the initialization of the depth values.…”

Realistic retargeting of facial video

“…A good survey can be found in [1]. This paper is related to marker-less facial motion capture [2,3,4,5,6] insofar as it uses related techniques to accurately track the head position in the video footage. For example, [3] uses optical flow together with a photogrammetric reconstruction to drive a cyberscan model of the actor's face in neutral expression in order to obtain a time-consistent animation mesh.…”