Are emojis processed like words?: Eye movements reveal the time course of semantic processing for emojified text

Abstract

Emojis have many functions that support reading. Most obviously, they convey semantic information and support reading comprehension (Lo, CyberPsychology & Behavior, 11[5], 595-597, 2008; Riordan, Computers in Human Behavior, 76, 75-86, 2017b). However, it is undetermined whether emojis recruit the same perceptual and cognitive processes for identification and integration during reading as do words. To investigate whether emojis are processed like words, we used eye tracking to examine the time course of semantic processing of emojis during reading. Materials consisted of sentences containing a target word (e.g., coffee in the sentence "My tall coffee is just the right temperature") when there was no emoji present and when there was a semantically congruent (i.e., synonymous) emoji (e.g., the cup of coffee emoji, ) or an incongruent emoji (e.g., the beer mug emoji, ) present at the end of the sentence. Similar to congruency effects with words, congruent emojis were fixated for shorter periods and were less likely to be refixated than were incongruent emojis. In addition, congruent emojis were more frequently skipped than incongruent emojis, which suggests that semantic aspects of emoji processing begin in the parafovea. Finally, the presence of an emoji, relative to its absence increased target-word skipping rates and reduced total time on target words. We discuss the implications of our findings for models of eye-movement control during reading.Keywords Emojis . Eye movements and reading . Reading models . Parafoveal processing Are emojis processed like words? As with words, a core function of emojis is to convey semantic information (Lo, 2008;Riordan, 2017b). Although emojis serve many different functions to support reading, it is unknown whether the same cognitive and perceptual processes that support the identification and integration of words during reading also extend to emojis. Given the growing popularity of emojis on the internet, social media (Novak, Smailović, Sluban, & Mozetič, 2015), and in text-based communication (Tigwell & Flatla, 2016; i.e., more than 3,000 emojis are available to date; Unicode 12.1; https:// unicode.org/emoji/charts/full-emoji-list.html), we aimed to determine whether the time course of semantic processing for emojis is similar to words. The time course of semantic processing provides empirical constraints for the development of models of eye movement control during reading (for reviews, see Rayner, 1998Rayner, , 2009. Considering the ubiquity * Eliza Barach

Proceedings of the The Fifth International Workshop on Emoji Understanding and Applications in Social Media

“…With respect to recognition accuracy, results show that, consistent with the levels of processing framework, readers had more accurate recognition memory for emojis that were semantically relevant to the paired text compared to emojis that were not related to the text, suggesting that readers encoded the semantic content of the emojis and integrated it with the surrounding text. The present study extends previous findings suggesting that readers show semantic congruency effects in online processing measures (Barach et al, 2021;Beyersmann et al, 2022) by providing insight about what is retained in memory shortly after reading passages with emojis. Specifically, our findings suggest that readers encode the semantic properties of emojis in memory and show better recognition of semantically congruent emojis compared to semanticallyincongruent emojis after a short delay.…”

“…Based on the levels of processing framework and prior work showing that readers show semantic congruency effects for emojis during reading (Barach et al, 2021;Beyersmann et al, 2022), we predicted that readers would show higher recognition accuracy for emojis that are congruent with the surrounding passage content (synonym and inference emojis) than incongruent emojis. Extrapolating from the levels of processing framework (Craik and Lockhart, 1972), we also hypothesized a memory advantage for emojis consistent with passage inferences, such that readers would show higher recognition accuracy for inference emojis than for synonym emojis.…”

“…Recent research using both ratings tasks and online processing measures has shown that emojis that are redundant (i.e., synonymous) with a target word can facilitate text comprehension (Daniel and Camp, 2020). Barach and colleagues (Barach et al, 2021) used eye tracking measures to examine how readers benefit from the presence of non-face emojis that were positioned at the end of sentences and were synonymous with the target word (e.g., coffee). They compared sentences with emojis that were either semantically congruent with the target word (e.g., "My tall coffee is just the right temperature "), semantically incongruent (e.g., "My tall coffee is just the right temperature "), and sentences without an emoji ("My tall coffee is just the right temperature").…”

That’s Good News ☹

“…However, the direct relationship between emoji congruency and reading remains less understood. To the best of our knowledge, only two prior eye-tracking studies have examined the role of emojis in sentence processing using eye-tracking (Barach et al, 2021; Robus et al, 2020). Robus et al (2020) investigated the effect of emoji valence on text valence but found no evidence for a relationship.…”

“…Robus et al (2020) investigated the effect of emoji valence on text valence but found no evidence for a relationship. Barach et al (2021) examined adults’ eye movements on sentences including non-face emojis in sentence final position. Each sentence included a target word ( coffee ) that was either congruent/synonymous ( My tall coffee is just the right temperature ☕) or incongruent with the emoji ( My tall coffee is just the right temperature 🍺), relative to a no-emoji control condition ( My tall coffee is just the right temperature ).…”