A Transfer Learning Approach to Heatmap Regression for Action Unit Intensity Estimation

Ntinou, Ioanna; Sánchez, Ernesto Sánchez; Bulat, Adrian; Valstar, Michel; Tzimiropoulos, Georgios

doi:10.1109/taffc.2021.3061605

Cited by 24 publications

(19 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, our AP brings the largest improvement setting a new state-of-the-art ICC of 0.58, which improves over the Heatmap Regression method of Ntinou et al [46]. Finally, we compare our method against the winner and runner-up methods of FERA 2015, ISIR [45] and CDL [1], respectively, as well as against the Heatmap Regression method of [46]. Our method sets up a new state-of-the-art result with an ICC score of 0.74.…”

Section: Modelmentioning

confidence: 78%

“…These labels are noisy, and lack temporal consistency, yet they still provide some information regarding the true labels. In this paper, we choose as our backbone an architecture inspired by that of [46,60,73]. We refer the reader to the Supplementary Material for further details.…”

Section: Input Features and Pseudo-labelsmentioning

confidence: 99%

“…On DISFA, our backbone is again on par with the state of the art methods VGP-AE [12] and 2DC [39]. However, our AP brings the largest improvement setting a new state-of-the-art ICC of 0.58, which improves over the Heatmap Regression method of Ntinou et al [46]. Finally, we compare our method against the winner and runner-up methods of FERA 2015, ISIR [45] and CDL [1], respectively, as well as against the Heatmap Regression method of [46].…”

Section: Modelmentioning

confidence: 99%

“…In this paper, we address the problem of facial behaviour recognition from video, in particular, the problem of recognising apparent emotions in terms of Valence and Arousal [51], and facial expressions in terms of Action Unit intensity [11]. This is a longstanding problem in video recognition which has been extensively studied by the computer vision community [26,64,41,21,33,28,67,74,77,71,55,53,54,39,46]. Nevertheless, even recent methods [27,33,77] struggle to achieve high accuracy on the most difficult datasets including SEWA [34], Aff-Wild2 [29], BP4D [75] and DISFA [42].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

Sánchez¹,

Tellamekala²,

Valstar³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Temporal context is key to the recognition of expressions of emotion. Existing methods, that rely on recurrent or selfattention models to enforce temporal consistency, work on the feature level, ignoring the task-specific temporal dependencies, and fail to model context uncertainty. To alleviate these issues, we build upon the framework of Neural Processes to propose a method for apparent emotion recognition with three key novel components: (a) probabilistic contextual representation with a global latent variable model; (b) temporal context modelling using task-specific predictions in addition to features; and (c) smart temporal context selection. We validate our approach on four databases, two for Valence and Arousal estimation (SEWA and AffWild2), and two for Action Unit intensity estimation (DISFA and BP4D). Results show a consistent improvement over a series of strong baselines as well as over state-ofthe-art methods.

show abstract

Section: Modelmentioning

confidence: 78%

Section: Input Features and Pseudo-labelsmentioning

confidence: 99%

Section: Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

Sánchez¹,

Tellamekala²,

Valstar³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, many researchers consider the work done by Ntinou et al [141] as the state-of-the-art method for jointly estimating landmark localization and AU intensity. In this work, researchers employed heatmap regression to model the the existence of an AU at specific location.…”

Section: Jointly Estimating Landmark Detection and Action Unit Intensitymentioning

confidence: 99%

Facial Expression Modeling and Synthesis for Patient Simulator Systems: Past, Present, and Future

Pourebadi

Riek

2022

ACM Trans. Comput. Healthcare

View full text Add to dashboard Cite

Clinical educators have used robotic and virtual patient simulator systems (RPS) for dozens of years, to help clinical learners (CL) gain key skills to help avoid future patient harm. These systems can simulate human physiological traits; however, they have static faces and lack the realistic depiction of facial cues, which limits CL engagement and immersion. In this article, we provide a detailed review of existing systems in use, as well as describe the possibilities for new technologies from the human–robot interaction and intelligent virtual agents communities to push forward the state of the art. We also discuss our own work in this area, including new approaches for facial recognition and synthesis on RPS systems, including the ability to realistically display patient facial cues such as pain and stroke. Finally, we discuss future research directions for the field.

show abstract

Semi-supervised Facial Action Unit Intensity Estimation with Contrastive Learning

Sánchez

Bulat

Zaganidis

et al. 2021

Computer Vision – ACCV 2020

View full text Add to dashboard Cite

A Transfer Learning Approach to Heatmap Regression for Action Unit Intensity Estimation

Cited by 24 publications

References 60 publications

Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

Facial Expression Modeling and Synthesis for Patient Simulator Systems: Past, Present, and Future

Semi-supervised Facial Action Unit Intensity Estimation with Contrastive Learning

Contact Info

Product

Resources

About