Can the crowd tell how I feel? Trait empathy and ethnic background in a visual pain judgment task

Abstract

Many advocate for artificial agents to be empathic. Crowdsourcing could help, by facilitating human-in-the-loop approaches and data set creation for visual emotion recognition algorithms. Although crowdsourcing has been employed successfully for a range of tasks, it is not clear how effective crowdsourcing is when the task involves subjective rating of emotions. We examined relationships between demographics, empathy, and ethnic identity in pain emotion recognition tasks. Amazon MTurkers viewed images of strangers in painful settings, and tagged subjects' emotions. They rated their level of pain arousal and confidence in their responses, and completed tests to gauge trait empathy and ethnic identity. We found that Caucasian participants were less confident than others, even when viewing other Caucasians in pain. Gender correlated to word choices for describing images, though not to pain arousal or confidence. The results underscore the need for verified information on crowdworkers, to harness diversity effectively for metadata generation tasks.

Social Cues, Social Biases: Stereotypes in Annotations on People Images

“…Indeed, we found systematic differences between responses of women and men contributors, which is in line not only with theory but also other recent findings on the crowdsourcing of image annotation. For instance, Matsangidou and colleagues (Matsangidou et al 2018) asked workers to describe images of people depicted in painful situations, as well as to evaluate their own levels of emotional arousal. While men and women reported similar levels of arousal when viewing the images, as well as comparable levels of confidence in the accuracy of their work, significant qualitative differences in the words used to describe the images were reported.…”