Implications for Emotion: Using Anatomically Based Facial Coding to Compare Emoji Faces Across Platforms

Abstract

Emoji faces, which are ubiquitous in our everyday communication, are thought to resemble human faces and aid emotional communication. Yet, few studies examine whether emojis are perceived as a particular emotion and whether that perception changes based on rendering differences across electronic platforms. The current paper draws upon emotion theory to evaluate whether emoji faces depict anatomical differences that are proposed to differentiate human depictions of emotion (hereafter, “facial expressions”). We modified the existing Facial Action Coding System (FACS) (Ekman and Rosenberg, 1997) to apply to emoji faces. An equivalent “emoji FACS” rubric allowed us to evaluate two important questions: First, Anatomically, does the same emoji face “look” the same across platforms and versions? Second, Do emoji faces perceived as a particular emotion category resemble the proposed human facial expression for that emotion? To answer these questions, we compared the anatomically based codes for 31 emoji faces across three platforms and two version updates. We then compared those codes to the proposed human facial expression prototype for the emotion perceived within the emoji face. Overall, emoji faces across platforms and versions were not anatomically equivalent. Moreover, the majority of emoji faces did not conform to human facial expressions for an emotion, although the basic anatomical codes were shared among human and emoji faces. Some emotion categories were better predicted by the assortment of anatomical codes than others, with some individual differences among platforms. We discuss theories of emotion that help explain how emoji faces are perceived as an emotion, even when anatomical differences are not always consistent or specific to an emotion.

Semantic differences in visually similar face emojis

“…Constraints were that each emoji should occur in only one pair and that we aimed to avoid emojis with symbolic components like or , which cannot reflect actual human faces. The annotation of the emojis' AUs, given in Figure 1, is based on the coding rubric by Fugate and Franco (2021). With the exception of the emojis grinning squinting face, astonished face, slightly frowning face and frowning face, which are not annotated in their paper, the AU codings were identical to Fugate and Franco's (2021) existing AU codings for the emojis.…”

“…The annotation of the emojis' AUs, given in Figure 1, is based on the coding rubric by Fugate and Franco (2021). With the exception of the emojis grinning squinting face, astonished face, slightly frowning face and frowning face, which are not annotated in their paper, the AU codings were identical to Fugate and Franco's (2021) existing AU codings for the emojis. Note, however, that Fugate and Franco do not provide AU codings for WhatsApp emojis.…”

“…Whether the difference between visually similar face emojis maps to a difference in corresponding human facial expressions can be captured via the Facial Action Coding System (FACS; Ekman & Friesen, 1978), which decomposes facial expressions into minimal Action Units (AUs) and has been adapted for the annotation of face emojis by Fugate and Franco (2021). Against the background of the FACS, we can distinguish two types of pairs of visually similar emojis: Emoji pairs with a visual difference that corresponds to an AU difference ([AU+]), like the pair and (grinning face with big eyes, grinning squinting face) and emoji pairs with a visual difference that does not correspond to an AU difference ([AUÀ]), like the pair and ( grinning face with smiling eyes, beaming face with smiling eyes).…”

“…The latter pair shares the AUs 12 + 25 + 26 + 63, representing an arched-up mouth with an open mouth space and eyes looking up, but does not differ in any other Action Units. We followed Fugate and Franco (2021) in coding the eyes as AU 63 'eyes looking up'. Alternatively, this eye shape could be interpreted as AU 6 'cheek raiser' in both emojis.…”

The recognition of emotions conveyed by emoticons and emojis: A systematic literature review.

“…For each emoji, the database provides a name, a code, and the different graphic charter available. In recent research, inspired by the FACS (Ekman et al, 2002), Fugate and Franco (2021) created an equivalent "emoji FACS" system attributing action units (AUs) to 31 graphic emoticons. They found a difference in AU count across the three main platforms (Apple, Google Android, and Samsung), and between different versions of graphic emoticons on a given platform.…”