Py-Feat: Python Facial Expression Analysis Toolbox

Cheong, Jin Hyun; Xie, Tiankang; Byrne, Sophie; Chang, Luke J.

doi:10.1007/s42761-023-00191-4

Cited by 28 publications

(11 citation statements)

References 91 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Spoken language samples are also acquired from audio diaries captured over smartphones. Face processing uses the open-source software packages Py-Feat 74 v0.5 and MediaPipe v0.9.3.0 75 to determine the number and location of faces in a video. The landmark detection algorithms in these packages support the extraction of facial action units, 76 allowing for the detection of common expressions and emotions.…”

Section: Methodsmentioning

confidence: 99%

Accelerating Medicines Partnership® Schizophrenia (AMP® SCZ): Rationale and Study Design of the Largest Global Prospective Cohort Study of Clinical High Risk for Psychosis

Wannan,

Nelson,

Addington

et al. 2024

Schizophrenia Bulletin

View full text Add to dashboard Cite

This article describes the rationale, aims, and methodology of the Accelerating Medicines Partnership® Schizophrenia (AMP® SCZ). This is the largest international collaboration to date that will develop algorithms to predict trajectories and outcomes of individuals at clinical high risk (CHR) for psychosis and to advance the development and use of novel pharmacological interventions for CHR individuals. We present a description of the participating research networks and the data processing analysis and coordination center, their processes for data harmonization across 43 sites from 13 participating countries (recruitment across North America, Australia, Europe, Asia, and South America), data flow and quality assessment processes, data analyses, and the transfer of data to the National Institute of Mental Health (NIMH) Data Archive (NDA) for use by the research community. In an expected sample of approximately 2000 CHR individuals and 640 matched healthy controls, AMP SCZ will collect clinical, environmental, and cognitive data along with multimodal biomarkers, including neuroimaging, electrophysiology, fluid biospecimens, speech and facial expression samples, novel measures derived from digital health technologies including smartphone-based daily surveys, and passive sensing as well as actigraphy. The study will investigate a range of clinical outcomes over a 2-year period, including transition to psychosis, remission or persistence of CHR status, attenuated positive symptoms, persistent negative symptoms, mood and anxiety symptoms, and psychosocial functioning. The global reach of AMP SCZ and its harmonized innovative methods promise to catalyze the development of new treatments to address critical unmet clinical and public health needs in CHR individuals.

show abstract

Section: Methodsmentioning

confidence: 99%

Accelerating Medicines Partnership® Schizophrenia (AMP® SCZ): Rationale and Study Design of the Largest Global Prospective Cohort Study of Clinical High Risk for Psychosis

Wannan,

Nelson,

Addington

et al. 2024

Schizophrenia Bulletin

View full text Add to dashboard Cite

show abstract

“…Considering that past studies have suggested that positive emotional signals on faces may enhance face recognition memory 212 in this study, the Python-based software tool Py-Feat was used to obtain a computational measure for displayed facial emotion signals 220 . This method uses a six emotion model 221 plus a neutral state.…”

Section: Methodsmentioning

confidence: 99%

Comparing episodic memory outcomes from walking augmented reality and stationary virtual reality encoding experiences

Pastor,

Bourdin-Kreitz

2024

Sci Rep

View full text Add to dashboard Cite

Episodic Memory (EM) is the neurocognitive capacity to consciously recollect personally experienced events in specific spatio-temporal contexts. Although the relevance of spatial and temporal information is widely acknowledged in the EM literature, it remains unclear whether and how EM performance and organisation is modulated by self-motion, and by motor- and visually- salient environmental features (EFs) of the encoding environment. This study examines whether and how EM is modulated by locomotion and the EFs encountered in a controlled lifelike learning route within a large-scale building. Twenty-eight healthy participants took part in a museum-tour encoding task implemented in walking Augmented Reality (AR) and stationary Virtual Reality (VR) conditions. EM performance and organisation were assessed immediately and 48-hours after trials using a Remember/Familiar recognition paradigm. Results showed a significant positive modulation effect of locomotion on distinctive EM aspects. Findings highlighted a significant performance enhancement effect of stairway-adjacent locations compared to dead-end and mid-route stimuli-presentation locations. The results of this study may serve as design criteria to facilitate neurocognitive rehabilitative interventions of EM. The underlying technological framework developed for this study represents a novel and ecologically sound method for evaluating EM processes in lifelike situations, allowing researchers a naturalistic perspective into the complex nature of EM.

show abstract

“…Py-Feat is an open-source Python toolbox [42]. Version 0.6.0 includes various pretrained models for each face-processing step.…”

Section: Py-featmentioning

confidence: 99%

“…It was trained for 20 AUs (1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 17, 18, 20, 23, 24, 25, 26, 28, 43) using BP4D [57], BP4D+ [68], DISFA [55], DISFA+ [69], CK+ [70], JAFFE [71], Shoulder Pain [58], and EmotioNet [72] and validated using WIDER FACE [51], 300W [73], NAMBA [24], and BIWI-Kinect [74]. The average F1 score was 0.54 (AU4 = 0.64 and AU12 = 0.83) [42].…”

Section: Py-featmentioning

confidence: 99%

“…(6) Previous studies validating the detection of spontaneous facial mimicry using the automated FACS employed earlier versions of commercial software, such as FaceReader 7 [33,34]. In this study, FaceReader 9.0, which incorporated updates such as deep learning-based algorithms (see Section 2.6.1) and two new open-source software, OpenFace 2.2.0 [41] and Py-Feat 0.6.0 [42], will be estimated. Section 1 provides the literature review in relation to the present study; Section 2 provides a comprehensive overview of the methods and materials; Section 3 presents the statistical results and visualization; and Section 4 discusses the research results, limitations, and future directions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding

Hsu,

Sato

2023

Sensors

View full text Add to dashboard Cite

Although electromyography (EMG) remains the standard, researchers have begun using automated facial action coding system (FACS) software to evaluate spontaneous facial mimicry despite the lack of evidence of its validity. Using the facial EMG of the zygomaticus major (ZM) as a standard, we confirmed the detection of spontaneous facial mimicry in action unit 12 (AU12, lip corner puller) via an automated FACS. Participants were alternately presented with real-time model performance and prerecorded videos of dynamic facial expressions, while simultaneous ZM signal and frontal facial videos were acquired. Facial videos were estimated for AU12 using FaceReader, Py-Feat, and OpenFace. The automated FACS is less sensitive and less accurate than facial EMG, but AU12 mimicking responses were significantly correlated with ZM responses. All three software programs detected enhanced facial mimicry by live performances. The AU12 time series showed a roughly 100 to 300 ms latency relative to the ZM. Our results suggested that while the automated FACS could not replace facial EMG in mimicry detection, it could serve a purpose for large effect sizes. Researchers should be cautious with the automated FACS outputs, especially when studying clinical populations. In addition, developers should consider the EMG validation of AU estimation as a benchmark.

show abstract

Py-Feat: Python Facial Expression Analysis Toolbox

Cited by 28 publications

References 91 publications

Accelerating Medicines Partnership® Schizophrenia (AMP® SCZ): Rationale and Study Design of the Largest Global Prospective Cohort Study of Clinical High Risk for Psychosis

Accelerating Medicines Partnership® Schizophrenia (AMP® SCZ): Rationale and Study Design of the Largest Global Prospective Cohort Study of Clinical High Risk for Psychosis

Comparing episodic memory outcomes from walking augmented reality and stationary virtual reality encoding experiences

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding

Contact Info

Product

Resources

About