Testing, explaining, and exploring models of facial expressions of emotions

Snoek, Lukas; Jack, Rachael E.; Schyns, Philippe G.; Garrod, Oliver; Mittenbühler, Maximilian; Chen, Chaona; Oosterwijk, Suzanne; Scholte, H. Steven

doi:10.1126/sciadv.abq8421

Cited by 8 publications

(3 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While many prior studies have evaluated facial action unit (AU) differences in relation to pain in controlled settings 46,47 , we found that facial expression changes underlying self-reported pain states had little overlap with conventionally reported pain-related AUs. Instead, these AUs (AU14, AU15, AU26) may reflect more general changes in affect 48 . In contrast, facial expressions associated with momentary pain exhibited many of the canonical pain-related AUs (AU5, AU6/7, AU9/10, AU11/12, AU25/26), suggesting immediate pain onset.…”

Section: Discussionmentioning

confidence: 99%

Naturalistic acute pain states decoded from neural and facial dynamics

Huang,

Gopal,

Kakusa

et al. 2024

Preprint

View full text Add to dashboard Cite

Pain is a complex experience that remains largely unexplored in naturalistic contexts, hindering our understanding of its neurobehavioral representation in ecologically valid settings. To address this, we employed a multimodal, data-driven approach integrating intracranial electroencephalography, pain self-reports, and facial expression quantification to characterize the neural and behavioral correlates of naturalistic acute pain in twelve epilepsy patients undergoing continuous monitoring with neural and audiovisual recordings. High self-reported pain states were associated with elevated blood pressure, increased pain medication use, and distinct facial muscle activations. Using machine learning, we successfully decoded individual participants' high versus low self-reported pain states from distributed neural activity patterns (mean AUC = 0.70), involving mesolimbic regions, striatum, and temporoparietal cortex. High self-reported pain states exhibited increased low-frequency activity in temporoparietal areas and decreased high-frequency activity in mesolimbic regions (hippocampus, cingulate, and orbitofrontal cortex) compared to low pain states. This neural pain representation remained stable for hours and was modulated by pain onset and relief. Objective facial expression changes also classified self-reported pain states, with results concordant with electrophysiological predictions. Importantly, we identified transient periods of momentary pain as a distinct naturalistic acute pain measure, which could be reliably differentiated from affect-neutral periods using intracranial and facial features, albeit with neural and facial patterns distinct from self-reported pain. These findings reveal reliable neurobehavioral markers of naturalistic acute pain across contexts and timescales, underscoring the potential for developing personalized pain interventions in real-world settings.

show abstract

Section: Discussionmentioning

confidence: 99%

Naturalistic acute pain states decoded from neural and facial dynamics

Huang,

Gopal,

Kakusa

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Facial expressions of humans can be classified into two types based on the methods used for emotion recognition: rule-based and machine learning-based. The first approach employs predefined rules and features to recognize and reproduce human facial expressions ( Snoek et al, 2023 ). It relies on expert-defined rules and models to infer emotional states by calculating facial features such as eye shape, mouth movement, and eyebrow position, as well as vocal features like intonation and speech rate.…”

Section: Progress In the Application Of Brain-machine Interfaces In T...mentioning

confidence: 99%

Advancements in brain-machine interfaces for application in the metaverse

Liu,

et al. 2024

Front. Neurosci.

View full text Add to dashboard Cite

In recent years, with the shift of focus in metaverse research toward content exchange and social interaction, breaking through the current bottleneck of audio-visual media interaction has become an urgent issue. The use of brain-machine interfaces for sensory simulation is one of the proposed solutions. Currently, brain-machine interfaces have demonstrated irreplaceable potential as physiological signal acquisition tools in various fields within the metaverse. This study explores three application scenarios: generative art in the metaverse, serious gaming for healthcare in metaverse medicine, and brain-machine interface applications for facial expression synthesis in the virtual society of the metaverse. It investigates existing commercial products and patents (such as MindWave Mobile, GVS, and Galea), draws analogies with the development processes of network security and neurosecurity, bioethics and neuroethics, and discusses the challenges and potential issues that may arise when brain-machine interfaces mature and are widely applied. Furthermore, it looks ahead to the diverse possibilities of deep and varied applications of brain-machine interfaces in the metaverse in the future.

show abstract

“…2. Accounting for Individual Differences: It is acknowledged that different individuals may use different compositions of facial AUs to perceive the same emotion 47,48 . To preserve such individual differences, the methodological and analytical approaches must be able to discern the specific AUs that each participant utilizes to perceive each emotion, instead of assuming what these AUs are, or building an average that represents no one.…”

Section: Introductionmentioning

confidence: 99%

The Brain Computes Dynamic Facial Movements for Emotion Categorization Using a Third Pathway

Yan,

Zhan,

Garrod

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Recent theories suggest a new brain pathway dedicated to processing social movement is involved in understanding emotions from biological motion, beyond the well-known ventral and dorsal pathways. However, how this social pathway functions as a network that computes dynamic biological motion signals for perceptual behavior is unchartered. Here, we used a generative model of important facial movements that participants (N = 10) categorized as "happy," "surprise," "fear," "anger," "disgust," "sad" while we recorded their MEG brain responses. Using new representational interaction measures (between facial features, MEGt source, and behavioral responses), we reveal per participant a functional social pathway extending from occipital cortex to superior temporal gyrus. Its MEG sources selectively represent, communicate and compose facial movements to disambiguate emotion categorization behavior, while occipital cortex swiftly filters out task-irrelevant identity-defining face shape features. Our findings reveal how social pathway selectively computes complex dynamic social signals to categorize emotions in individual participants.

show abstract

Testing, explaining, and exploring models of facial expressions of emotions

Cited by 8 publications

References 73 publications

Naturalistic acute pain states decoded from neural and facial dynamics

Naturalistic acute pain states decoded from neural and facial dynamics

Advancements in brain-machine interfaces for application in the metaverse

The Brain Computes Dynamic Facial Movements for Emotion Categorization Using a Third Pathway

Contact Info

Product

Resources

About