Gaze patterns reveal how situation models and text representations contribute to episodic text memory

Johansson, Roger; Oren, Franziska; Holmqvist, Kenneth

doi:10.1016/j.cognition.2018.02.016

Cited by 5 publications

(5 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LAN has been intensively studied in relation to memory retrieval (for example, Altmann, 2004; Altmann & Kamide, 2009; Bone et al, 2019; de Groot et al, 2016; Foerster, 2018; Johansson et al, 2012; Johansson et al, 2006; Johansson & Johansson, 2014, 2020; Johansson et al, 2018; Jones et al, 2018; Kinjo et al, 2020; Kumcu & Thompson, 2020; Laeng et al, 2014; Martarelli & Mast, 2011, 2013; Richardson & Kirkham, 2004; Richardson & Spivey, 2000; Scholz et al, 2018; Scholz et al, 2016; Spivey & Geng, 2001; Wantz et al, 2016; Wynn et al, 2020). In an experiment by Richardson and Spivey (2000), participants listened to semantic statements while a visual symbol appeared in one spatial location on a computer screen.…”

Section: Lan and Memory Retrievalmentioning

confidence: 99%

When the eyes have it and when not: How multiple sources of activation combine to guide eye movements during multiattribute decision making.

Rosner¹,

Schaffner²,

Helversen³

2022

Journal of Experimental Psychology: General

View full text Add to dashboard Cite

Memory plays a major but underexplored role in judgment and decision making (JDM). Studying eye movements-especially how people look at empty spatial locations when retrieving from memory information previously associated with those locations-provides useful information about how memory influences JDM. This so-called looking-at-nothing behavior is thought to reflect memory-driven allocation of attention. However, eye movements are also guided toward salient visual stimuli, such as test items presented on a screen. It is unclear how these multiple sources of activation combine to guide looking-at-nothing in JDM. We investigated this question in two experiments in which participants solved multiattribute categorization tasks using an exemplar-based decision strategy. In the first experiment, we tested how the occurrence and the strength of looking-at-nothing vary with the presentation format and the amount of training participants received. Looking-at-nothing occurred during categorizations when test-item information was presented auditorily and visually, but for the latter only after visual information was removed from the screen. It occurred both when training items were learned by heart and when they were presented 10 times on the screen. A second experiment revealed that an explicit instruction to imagine retrieval-relevant information during categorizations increased lookingat-nothing but did not change the decision-making process. The results shed light on the interaction between eye movements and attention to information in memory during JDM that can be explained in light of a shared priority map in memory. A detailed understanding of this interaction forms the basis for using eye movements to study memory processes in JDM.

show abstract

Section: Lan and Memory Retrievalmentioning

confidence: 99%

When the eyes have it and when not: How multiple sources of activation combine to guide eye movements during multiattribute decision making.

Rosner¹,

Schaffner²,

Helversen³

2022

Journal of Experimental Psychology: General

View full text Add to dashboard Cite

show abstract

“…It is worth noting that the effect of narrative specificity was specific to imageability and did not significantly influence overall comprehension of the verbal description. This implies that the improved ability to imagine spatiotemporal properties within a narrative is not directly related to a general understanding of the verbal content (cf., Holsanova, 2022), highlighting the importance of considering how the specific situation model -which goes beyond words, grammar, and propositions (e.g., Bergen, et al, 2004;Zwaan & Radvansky, 1998;Johansson et al, 2018) -is communicated in AD (Holsanova, 2016;2022).…”

Section: Discussionmentioning

confidence: 99%

“…Substantial neurocognitive research has established that comparable cognitive processes are activated when individuals internally simulate an event and when they directly perceive the same event (e.g., Johansson, Holsanova, & Holmqvist, 2006;Kosslyn et al, 2006;Peason, 2019). Thus, to grasp a specific situation conveyed within a spoken narration, listeners engage in mental imagery processes corresponding to a "situation model" (Van Dijk & Kintsch, 1983;Zwaan & Radvansky, 1998) of the described state-ofaffairs (Bergen, Chang, & Narayan, 2004;Johansson, Oren, & Holmqvist, 2018;Stanfield & Zwaan, 2001;Zwaan, Stanfield, & Yaxley, 2002). The capacity to create such situation models has been empirically demonstrated to wield substantial influence over ongoing verbal comprehension (Gambrell & Jawitz, 1993;Garnham, 1981;McKoon & Ratcliff, 1992), enabling receivers of verbal narratives to vicariously immerse themselves in the depicted content (Zwaan, 2004;Zwaan & Madden, 2009).…”

Section: Mental Imagery and Verbal Narrativesmentioning

confidence: 99%

The Influence of Narrative Specificity and Voice Quality when Listening to Audio Descriptions: a Comparison of the Sighted and the Blind

Åhlander,

Holsanova,

Johansson

2023

Preprint

View full text Add to dashboard Cite

Audio description (AD) serves as a vital means to make visual media accessible to non-sighted and visually impaired audiences. This study systematically investigates the impact of narrative specificity and voice quality on imageability and comprehension in both sighted and non-sighted populations. Twenty non-sighted participants, including congenitally blind individuals and those who lost their sight early in life, were compared with a group of 20 sighted participants, matched for verbal working memory capabilities. Participants listened to 50 short event descriptions, describing spatiotemporal relations with varying levels of narrative specificity, presented in both typical and dysphonic voices. After each event description, participants rated their ability to imagine the content, overall comprehension, listening effort, and listening enjoyment. Results indicate that high narrative specificity enhanced imageability in non-sighted individuals, especially for scenarios involving changes in motion, and to some extent, for visuospatial relations, irrespective of sightedness. Additionally, dysphonic voices increased listening effort and reduced enjoyment for non-sighted participants only. These findings underscore the importance of considering voice quality and narrative specificity in AD for non-sighted users and have implications for both professional audio describers and the development of automated AD systems.

show abstract

“…9 gaze metrics in AOIs [11,17] were received; however, due to the presence of their aliased values we considered only 5 metrics, Dwell Time (DT), First Fixation Duration (FFD), Fixation Count (FC), Fixation Time (FT), and Average Fixation Duration (AFD). 15 regression tests in JAMOVI (since there were 3 Event construal groups and 5 gaze metrics) and 30 tests (15 with Text AOIs and 15 with Image AOIs) were performed with monomodal and cross-modal switches, respectively.…”

Section: Data Preparationmentioning

confidence: 99%

Gaze behavior in single-page monomodal and cross-modal switches as affected by Event construal

Kiose¹,

Rzheshevskaya²,

Izmalkova³

2022

Computational Linguistics and Intellectual Technologies

View full text Add to dashboard Cite

This paper proposes Event-modelling Framework (EMF) to explore the systemic Event construal effects onto gaze. In two eye tracking experiments examining monomodal and cross-modal switches with a total number of 898 gaze probes of Text and Picture Areas of Interest (AOIs), we investigated gaze behavior variations produced by Event construal registered in 61 parameters of Referent, Event Frame and Perspective construal. 5 non-aliased gaze metrics served to assess the significance of Event construal effects and scale them. The results contribute to the research field of ergonomic optimization in visual and reading tasks.

show abstract

Gaze patterns reveal how situation models and text representations contribute to episodic text memory

Cited by 5 publications

References 78 publications

When the eyes have it and when not: How multiple sources of activation combine to guide eye movements during multiattribute decision making.

When the eyes have it and when not: How multiple sources of activation combine to guide eye movements during multiattribute decision making.

The Influence of Narrative Specificity and Voice Quality when Listening to Audio Descriptions: a Comparison of the Sighted and the Blind

Gaze behavior in single-page monomodal and cross-modal switches as affected by Event construal

Contact Info

Product

Resources

About