Realtime facial animation with on-the-fly correctives

Li, Hao; Yu, Jihun; Ye, Yuting; Bregler, Christoph

doi:10.1145/2461912.2462019

Cited by 262 publications

(209 citation statements)

References 46 publications

Supporting

Mentioning

209

Contrasting

Order By: Relevance

“…We detect dynamic occlusions caused by temporal shape and texture variations using an outlier voting scheme in superpixel space. As recently demonstrated by Li et al [31], the combination of sparse 2D facial features (e.g., eyes, eyebrows, and mouth) with dense depth maps are particularly effective in improving tracking fidelity. However, because facial landmark detection becomes significantly less reliable when the face is occluded, we synthesize plausible face textures right after our face segmentation step.…”

Section: Introductionmentioning

confidence: 89%

“…The data-driven method of Weise et al [39] uses a motion prior database to handle noise and the lowresolution depth maps from the Kinect sensor. For improved fidelity, techniques that combine depth input data with sparse facial features were introduced [31,11,13,9]. To improve accessibility with less input training, an example-based facial rigging method was introduced by Li et al [30].…”

Section: Related Workmentioning

confidence: 99%

“…To improve accessibility with less input training, an example-based facial rigging method was introduced by Li et al [30]. The method of [31] builds a single neutral model before tracking and trains PCA-based correctives for the expressions during tracking with samples obtained from per-vertex Laplacian deformations. Bouaziz et al [6] introduce a completely calibration-free system by modeling the full blendshape model during tracking.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Unconstrained realtime facial performance capture

Hsieh

et al. 2015

2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

104

View full text Add to dashboard Cite

We introduce a realtime facial tracking system specifically designed for performance capture in unconstrained settings using a consumer-level RGB-D sensor. Our framework provides uninterrupted 3D facial tracking, even in the presence of extreme occlusions such as those caused by hair, hand-to-face gestures, and wearable accessories. Anyone's face can be instantly tracked and the users can be switched without an extra calibration step. During tracking, we explicitly segment face regions from any occluding parts by detecting outliers in the shape and appearance input using an exponentially smoothed and user-adaptive tracking model as prior. Our face segmentation combines depth and RGB input data and is also robust against illumination changes. To enable continuous and reliable facial feature tracking in the color channels, we synthesize plausible face textures in the occluded regions. Our tracking model is personalized on-the-fly by progressively refining the user's identity, expressions, and texture with reliable samples and temporal filtering. We demonstrate robust and high-fidelity facial tracking on a wide range of subjects with highly incomplete and largely occluded data. Our system works in everyday environments and is fully unobtrusive to the user, impacting consumer AR applications and surveillance.

show abstract

Section: Introductionmentioning

confidence: 89%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Unconstrained realtime facial performance capture

Hsieh

et al. 2015

2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

104

View full text Add to dashboard Cite

show abstract

“…These techniques, however, cannot handle large deformations, and are not very practical for real-time applications. Real-time non-rigid reconstruction approaches have been achieved with the help of a template which is first acquired then used for tracking of non-rigidities with a good flexibility [40], [41]. Recently, we have proposed KinectDeform [11], the first nonrigid version of KinectFusion.…”

Section: Dynamic Multi-frame Approachesmentioning

confidence: 99%

Real-Time Enhancement of Dynamic Depth Videos with Non-Rigid Deformations

Ismaeil

Aouada

Solignac

et al. 2017

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Abstract-We propose a novel approach for enhancing depth videos containing non-rigidly deforming objects. Depth sensors are capable of capturing depth maps in real-time but suffer from high noise levels and low spatial resolutions. While solutions for reconstructing 3D details in static scenes, or scenes with rigid global motions have been recently proposed, handling unconstrained non-rigid deformations in relative complex scenes remains a challenge. Our solution consists in a recursive dynamic multi-frame superresolution algorithm where the relative local 3D motions between consecutive frames are directly accounted for. We rely on the assumption that these 3D motions can be decoupled into lateral motions and radial displacements. This allows to perform a simple local per-pixel tracking where both depth measurements and deformations are dynamically optimized. The geometric smoothness is subsequently added using a multi-level L 1 minimization with a bilateral total variation regularization. The performance of this method is thoroughly evaluated on both real and synthetic data. As compared to alternative approaches, the results show a clear improvement in reconstruction accuracy and in robustness to noise, to relative large non-rigid deformations, and to topological changes. Moreover, the proposed approach, implemented on a CPU, is shown to be computationally efficient and working in real-time.

show abstract

“…The application-specific nature of these approaches enables their authors to show excellent performance by taking advantage of domain-specific features and constraints, but it also prevents them from serving as general tools for tracking arbitrary articulated objects. Techniques have also been developed to track fully non-rigid deformations of an underlying surface template for both specific [21] and general [14,20,28] object cases. However, the full generality of these models comes at the cost of increased model complexity, and for many objects that are well modelled as piecewise rigid bodies, such overparameterized output obscures the utility of tracking the articulated body state directly.…”

Section: Introductionmentioning

confidence: 99%

DART: Dense Articulated Real-Time Tracking

Schmidt

Newcombe

Fox

2014

Robotics: Science and Systems X

137

130

View full text Add to dashboard Cite

Abstract-This paper introduces DART, a general framework for tracking articulated objects composed of rigid bodies connected through a kinematic tree. DART covers a broad set of objects encountered in indoor environments, including furniture and tools, and human and robot bodies, hands and manipulators. To achieve efficient and robust tracking, DART extends the signed distance function representation to articulated objects and takes full advantage of highly parallel GPU algorithms for data association and pose optimization. We demonstrate the capabilities of DART on different types of objects that have each required dedicated tracking techniques in the past. I. INTRODUCTIONThe ability to accurately track the pose of objects in real time is of fundamental importance to many areas of robotics. Applications range from navigation to planning, manipulation and human-robot interaction, all of which have received the attention of researchers working within a state-space modelbased paradigm within both computer vision and robotics. The class of objects that can be described as collections of rigid bodies chained together through a kinematic tree is quite broad, including furniture, tools, human bodies, human hands, and robot manipulators. Tracking articulated bodies from a single viewpoint and without instrumenting the object of interest still presents a significant challenge where the single viewpoint and occlusions, including self-occlusion, limit the amount of information available for pose estimation. Noisy sensor data and approximate object models pose additional problems. Finally, the objects being tracked can be highly dynamic and have many degrees of freedom, making real-time tracking difficult.Early articulated model-based tracking techniques relied on tracking 2D features such as image edges on a CPU [8,4]. Recently introduced depth cameras along with highly parallel algorithms optimized for modern GPUs have enabled new algorithms for tracking complex 3D objects in real time. Examples include KinectFusion and related efforts for 3D mapping [23,16,34], human body pose tracking [29,35,15], articulated hand tracking [24,19,26]. These approaches were developed for specific application domains and have not been demonstrated or tested on multiple tracking applications. The application-specific nature of these approaches enables their authors to show excellent performance by taking advantage of domain-specific features and constraints, but it also prevents them from serving as general tools for tracking arbitrary articulated objects. Techniques have also been developed to

show abstract

Realtime facial animation with on-the-fly correctives

Cited by 262 publications

References 46 publications

Unconstrained realtime facial performance capture

Unconstrained realtime facial performance capture

Real-Time Enhancement of Dynamic Depth Videos with Non-Rigid Deformations

DART: Dense Articulated Real-Time Tracking

Contact Info

Product

Resources

About