Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels

Washington, Peter; Kalantarian, Haik; Kent, Jack; Husic, Arman; Kline, Aaron; Leblanc, Émilie; Hou, Cathy; Mutlu, Cezmi; Dunlap, Kaitlyn; Penev, Yordan; Chrisman, Brianna; Paskov, Kelley; Jung, Jae-Yoon; Voss, Catalin; Haber, Nick; Wall, Dennis P.

doi:10.1007/s12559-021-09936-4

Cited by 18 publications

(13 citation statements)

References 94 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our plan is to develop the STAND app into a versatile tool that can be customized for researching and gathering computer vision data of the face to build personalized machine learning models which can support digital health interventions related to mental states and developmental disorders. This work can integrate with existing research in using automatic emotion recognition for a variety of contexts, including mental illness diagnosis, recognizing human social and physiological interactions, and developing sociable robotics and other human-computer interaction systems [57, [85][86][87][88][89]. For example, emotional expressions have a crucial role in recognizing certain types of developmental disorders.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Computer Vision Estimation of Stress and Anxiety Using a Gamified Mobile-based Ecological Momentary Assessment and Deep Learning: Research Protocol

Kargarandehkordi¹,

Washington²

2023

Preprint

Self Cite

View full text Add to dashboard Cite

Stress and anxiety can contribute to the development of major health problems such as heart disease, depression, and obesity. Due to its subjective nature, it is challenging to precisely measure human affect by relying on automated approaches. We therefore propose a personalized prediction framework fine-tuned for each participant in lieu of the traditional "one-size-fits-all" machine learning approach. We aim to collect such individualized data via two distinct procedures: 1) a smartphone-based ecological momentary assessment of stress, and 2) Zoom calls. The data collected from these periodic self-reports will include selfie photographs and ecological momentary assessments of affect. To enhance user engagement during the data collection process, we propose the use of gamification, an emerging trend which involves influencing user behavior and lifestyle by incorporating fun and engaging game elements into non-game contexts (e.g., health-related tasks). In addition to developing a standardized platform to achieve more accurate recognition of stress and anxiety, we plan to conduct a concurrent study in which we will capture videos of our subjects undertaking the Stroop Color Word and Amygdala Test and analyze the footage to identify additional significant characteristics that relate to anxiety. This could include features such as head and mouth movements, lip and cheek deformations, eye gaze, and blinking rates. The final results will provide a comparative evaluation of both objective measures of stress. This research project was approved by the University of Hawaii Institutional Review Board.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Computer Vision Estimation of Stress and Anxiety Using a Gamified Mobile-based Ecological Momentary Assessment and Deep Learning: Research Protocol

Kargarandehkordi¹,

Washington²

2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Previous work examined the use of crowdsourced annotations for autism, indicating that similar approaches could perhaps be applied through audio [31,[46][47][48][49][50][51]. Audio feature extraction combined with other autism classifiers could be used to create an explainable diagnostic system [52][53][54][55][56][57][58][59][60][61][62][63][64] fit for mobile devices [60]. Previous work investigated using such classifiers to detect autism or approach autism-related tasks like identifying emotion to improve socialization skills; combining computer vision-based quantification of relevant areas of interest, including hand stimming [58], upper limb movement [63], and eye contact [62,64], could possibly result in interpretable models.…”

Section: Future Workmentioning

confidence: 99%

Classifying Autism From Crowdsourced Semistructured Speech Recordings: Machine Learning Model Comparison Study

Chi¹,

Washington²,

Kline³

et al. 2022

JMIR Pediatr Parent

Self Cite

View full text Add to dashboard Cite

Background Autism spectrum disorder (ASD) is a neurodevelopmental disorder that results in altered behavior, social development, and communication patterns. In recent years, autism prevalence has tripled, with 1 in 44 children now affected. Given that traditional diagnosis is a lengthy, labor-intensive process that requires the work of trained physicians, significant attention has been given to developing systems that automatically detect autism. We work toward this goal by analyzing audio data, as prosody abnormalities are a signal of autism, with affected children displaying speech idiosyncrasies such as echolalia, monotonous intonation, atypical pitch, and irregular linguistic stress patterns. Objective We aimed to test the ability for machine learning approaches to aid in detection of autism in self-recorded speech audio captured from children with ASD and neurotypical (NT) children in their home environments. Methods We considered three methods to detect autism in child speech: (1) random forests trained on extracted audio features (including Mel-frequency cepstral coefficients); (2) convolutional neural networks trained on spectrograms; and (3) fine-tuned wav2vec 2.0—a state-of-the-art transformer-based speech recognition model. We trained our classifiers on our novel data set of cellphone-recorded child speech audio curated from the Guess What? mobile game, an app designed to crowdsource videos of children with ASD and NT children in a natural home environment. Results The random forest classifier achieved 70% accuracy, the fine-tuned wav2vec 2.0 model achieved 77% accuracy, and the convolutional neural network achieved 79% accuracy when classifying children’s audio as either ASD or NT. We used 5-fold cross-validation to evaluate model performance. Conclusions Our models were able to predict autism status when trained on a varied selection of home audio clips with inconsistent recording qualities, which may be more representative of real-world conditions. The results demonstrate that machine learning methods offer promise in detecting autism automatically from speech without specialized equipment.

show abstract

“…We considered, in particular, three different representations of ratings, which include some of the most common label representation approaches in the FER literature (Ko, 2018; Washington et al, 2021): Multi-label representation: each rater judgement is expressed in terms of the emotion(s) that they considered more intense, discarding the other less intense ones. For instance, for rater i, the list of emotions ‘surprise’ and ‘enjoyment’. Distribution-based representation: each rater judgement is represented in percentage terms of a whole constituted by all the non-absent emotions, like, for rater i the list: ‘enjoyment’: 67%’, ‘surprise’: 23%’, ‘anger’: 10%’. Ordinal representation: each rater judgement is simply expressed as the list of reported emotional intensity values between 1 and 5. Clearly, the multi-label representation corresponds to a categorical emotion model, while the ordinal and distribution-based representations take into account features of both categorical and dimensional emotion models.…”

Section: User Study: Methodsmentioning

confidence: 99%

Section: First Experiment: Inter-rater Reliabilitymentioning

confidence: 99%

The unbearable (technical) unreliability of automated facial emotion recognition

2022

View full text Add to dashboard Cite

Emotion recognition, and in particular acial emotion recognition (FER), is among the most controversial applications of machine learning, not least because of its ethical implications for human subjects. In this article, we address the controversial conjecture that machines can read emotions from our facial expressions by asking whether this task can be performed reliably. This means, rather than considering the potential harms or scientific soundness of facial emotion recognition systems, focusing on the reliability of the ground truths used to develop emotion recognition systems, assessing how well different human observers agree on the emotions they detect in subjects’ faces. Additionally, we discuss the extent to which sharing context can help observers agree on the emotions they perceive on subjects’ faces. Briefly, we demonstrate that when large and heterogeneous samples of observers are involved, the task of emotion detection from static images crumbles into inconsistency. We thus reveal that any endeavour to understand human behaviour from large sets of labelled patterns is over-ambitious, even if it were technically feasible. We conclude that we cannot speak of actual accuracy for facial emotion recognition systems for any practical purposes.

show abstract

Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels

Cited by 18 publications

References 94 publications

Computer Vision Estimation of Stress and Anxiety Using a Gamified Mobile-based Ecological Momentary Assessment and Deep Learning: Research Protocol

Computer Vision Estimation of Stress and Anxiety Using a Gamified Mobile-based Ecological Momentary Assessment and Deep Learning: Research Protocol

Classifying Autism From Crowdsourced Semistructured Speech Recordings: Machine Learning Model Comparison Study

The unbearable (technical) unreliability of automated facial emotion recognition

Contact Info

Product

Resources

About