A Comparison of Humans and Machine Learning Classifiers Detecting Emotion from Faces of People with Different Coverings

Abstract

Partial face coverings such as sunglasses and facemasks have now become the ‘new norm’, especially since the increase of infectious diseases. Unintentionally, they obscure facial expressions. Therefore, humans and artificial systems have been found to be less accurate in emotion categorization. However, it is unknown how similar the performance of humans compared with artificial systems is affected based on the exact same stimuli, varying systematically in types of coverings. Such a systematic direct comparison would allow conclusions about the relevant facial features in a naturalistic context. Therefore, we investigated the impact of facemasks and sunglasses on the ability to categorize emotional facial expressions in humans and artificial systems. Artificial systems, represented by the VGG19 deep learning algorithm, and humans assessed images of people with varying emotional facial expressions and with four different types of coverings, i.e. unmasked (original images), mask (mask covering lower-face), partial mask (with transparent mouth window), and sunglasses. Artificial systems performed significantly better than humans when no covering is present (> 15% difference). However, the achieved accuracy of both humans and artificial systems differed significantly depending on the type of coverings and, importantly, emotion, e.g. the use of sunglasses reduced accuracy for recognition of fear in humans. It was also noted that while humans mainly classify unknown expressions as neutral across all coverings, the misclassification varied in the artificial systems. These findings show humans and artificial systems classify and misclassify various emotion expressions differently depending on both the type of face covering and type of emotion.

A practical test of the link between perceived identifiability and prosociality with two field studies

“…The years 2020 and 2021 have been severely affected by the COVID-19 outbreak, and covering the face with masks is recommended and oftentimes mandatory in public settings to reduce the risk of transmission of the virus. With the possibility that face masks could remain highly present in social contexts in post-COVID-19 years, the health, social and psychological Face masks and prosocial behavior 5 consequences of face masks have been topics of interest in public debates and in research (e.g., Barrick et al, 2020;Beltran et al, 2021;Crandall & Bahns, 2021;Fatfouta & Trope, 2021;Grahlow et al, 2021;Shehu et al, 2021;Stajduhar et al, 2021;Williams et al, 2021).…”