Dense 3D face alignment from 2D videos in real-time

Jeni, László A.; Cohn, Jeffrey F.; Kanade, Takeo

doi:10.1109/fg.2015.7163142

Cited by 155 publications

(72 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because AAMs must be pre-trained for each participant, they are not well suited for clinical applications or large numbers of participants. We used a fully automatic, person-independent, generic approach that is comparable to AAMs to track the face and facial features, referred to as ZFace [45]. The robustness of this method for 3D registration and reconstruction from 2D video has been validated in a series of experiments (for details, see [45], [46]).…”

Section: Audiovisual Feature Extractionmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding

Dibeklioğlu

Hammal

Cohn

2018

IEEE J. Biomed. Health Inform.

Self Cite

151

View full text Add to dashboard Cite

Depression is one of the most common psychiatric disorders worldwide, with over 350 million people affected. Current methods to screen for and assess depression depend almost entirely on clinical interviews and self-report scales. While useful, such measures lack objective, systematic, and efficient ways of incorporating behavioral observations that are strong indicators of depression presence and severity. Using dynamics of facial and head movement and vocalization, we trained classifiers to detect three levels of depression severity. Participants were a community sample diagnosed with major depressive disorder. They were recorded in clinical interview (Hamilton Rating Scale for Depression, HRSD) at 7-week intervals over a period of 21 weeks. At each interview, they were scored by HRSD as moderately to severely depressed, mildly depressed, or remitted. Logistic regression classifiers using leave-one-participant-out validation were compared for facial movement, head movement, and vocal prosody individually and in combination. Accuracy of depression severity measurement from facial movement dynamics was higher than that for head movement dynamics; and each was substantially higher than that for vocal prosody. Accuracy using all three modalities combined only marginally exceeded that of face and head combined. These findings suggest that automatic detection of depression severity from behavioral indicators in patients is feasible and that multimodal measures afford most powerful detection.

show abstract

Section: Audiovisual Feature Extractionmentioning

confidence: 99%

“…This is done using a combined 3D supervised descent method [47], where the shape model is defined by a 3D mesh and the 3D vertex locations of the mesh [45]. ZFace registers a dense parameterized shape model to an image such that its landmarks correspond to consistent locations on the face.…”

Section: Audiovisual Feature Extractionmentioning

confidence: 99%

Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding

Dibeklioğlu

Hammal

Cohn

2018

IEEE J. Biomed. Health Inform.

Self Cite

151

View full text Add to dashboard Cite

show abstract

“…ZFace [18], a fully person-independent, generic approach, was used to track the registered face image. For each video frame, the tracker output the 3D coordinates of 49 fiducial points and 6 degrees of freedom of rigid head movement or a failure message when a frame could not be tracked (see Figure 3).…”

Section: Methodsmentioning

confidence: 99%

Automatic action unit detection in infants using convolutional neural network

Hammal

Chu

Cohn

et al. 2017

2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)

Self Cite

View full text Add to dashboard Cite

Action unit detection in infants relative to adults presents unique challenges. Jaw contour is less distinct, facial texture is reduced, and rapid and unusual facial movements are common. To detect facial action units in spontaneous behavior of infants, we propose a multi-label Convolutional Neural Network (CNN). Eighty-six infants were recorded during tasks intended to elicit enjoyment and frustration. Using an extension of FACS for infants (Baby FACS), over 230,000 frames were manually coded for ground truth. To control for chance agreement, inter-observer agreement between Baby-FACS coders was quantified using free-margin kappa. Kappa coefficients ranged from 0.79 to 0.93, which represents high agreement. The multi-label CNN achieved comparable agreement with manual coding. Kappa ranged from 0.69 to 0.93. Importantly, the CNN-based AU detection revealed the same change in findings with respect to infant expressiveness between tasks. While further research is needed, these findings suggest that automatic AU detection in infants is a viable alternative to manual coding of infant facial expression.

show abstract

“…The tracker registers a dense parameterized shape model to an image such that its landmarks correspond to consistent locations on the face. The robustness of the method for 3D registration and reconstruction from 2D video was validated in a series of experiments [for details, please see Jeni et al (2015)]. …”

Section: Automatic Tracking Of Head Orientation and Facial Landmarksmentioning

confidence: 99%

“…Occlusion, due to hands in mouth and extreme head movement, is another common problem. To overcome these challenges, we used a newly developed technique to track and align 3D features from 2D video (Jeni et al, 2015). This allowed us to ask to what extent head and facial movements communicate infants' affect.…”

mentioning

confidence: 99%

Automatic Measurement of Head and Facial Movement for Analysis and Detection of Infants’ Positive and Negative Affect

et al. 2015

Self Cite

View full text Add to dashboard Cite

Previous work in automatic affect analysis (AAA) has emphasized static expressions to the neglect of the dynamics of facial movement and considered head movement only a nuisance variable to control. We investigated whether the dynamics of head and facial movements apart from specific facial expressions communicate affect in infants, an under-studied population in AAA. Age-appropriate tasks were used to elicit positive and negative affect in 31 ethnically diverse infants. 3D head and facial movements were tracked from 2D video. Head angles in the horizontal (pitch), vertical (yaw), and lateral (roll) directions were used to measure head movement; and the 3D coordinates of 49 facial points to measure facial movements. Strong effects were found for both head and facial movements. Angular velocity and angular acceleration of head pitch, yaw, and roll were higher during negative relative to positive affect. Amplitude, velocity, and acceleration of facial movement were higher as well during negative relative to positive affect. A linear discriminant analysis using head and facial movement achieved a mean classification rate of positive and negative affect equal to 65% (Kappa = 0.30). Head and facial movements individually and in combination were also strongly related to observer ratings of affect intensity. Our results suggest that the dynamics of head and facial movements communicate affect at ages as young as 13 months. These interdisciplinary findings from behavioral science and computer vision deepen our understanding of communication of affect and provide a basis for studying individual differences in emotion in socio-emotional development.

show abstract

Dense 3D face alignment from 2D videos in real-time

Cited by 155 publications

References 38 publications

Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding

Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding

Automatic action unit detection in infants using convolutional neural network

Automatic Measurement of Head and Facial Movement for Analysis and Detection of Infants’ Positive and Negative Affect

Contact Info

Product

Resources

About