To search or to like: Mapping fixations to differentiate two forms of incidental scene memory

Abstract

We employed eye-tracking to investigate how performing different tasks on scenes (e.g., intentionally memorizing them, searching for an object, evaluating aesthetic preference) can affect eye movements during encoding and subsequent scene memory. We found that scene memorability decreased after visual search (one incidental encoding task) compared to intentional memorization, and that preference evaluation (another incidental encoding task) produced better memory, similar to the incidental memory boost previously observed for words and faces. By analyzing fixation maps, we found that although fixation map similarity could explain how eye movements during visual search impairs incidental scene memory, it could not explain the incidental memory boost from aesthetic preference evaluation, implying that implicit mechanisms were at play. We conclude that not all incidental encoding tasks should be taken to be similar, as different mechanisms (e.g., explicit or implicit) lead to memory enhancements or decrements for different incidental encoding tasks.

Overt attentional correlates of scene memorability and their relationships to scene semantics

“…First, the Edinburgh dataset (Luke et al, 2014;Nuthmann & Henderson, 2010;Pajak & Nuthmann, 2013), which has been used in prior publications (Choe et al, 2017;Einhäuser & Nuthmann, 2016;Kardan et al, 2015Kardan et al, , 2016Nuthmann, 2017) and is available from the author J.M.H. upon request, has the fixation map patterns of 135 scenes under three different encoding tasks (i.e., intentional memorization, visual search, and aesthetic preference evaluation) from 72 participants and memorability scores of these scenes from a subset of the participants (36).…”

“…Previous research has shown that intrinsic features of a scene, such as its semantics, global descriptors, objects counts or areas, interestingness, and aesthetics, can affect scene memorability in a similar manner across different viewers (Isola et al, 2011). However, scene memory can also be modulated by factors extrinsic to a scene, such as the other scenes that were presented with it (Bylinskii et al, 2015) and the viewing tasks that were performed while each scene was presented (Choe et al, 2017;Wolfe et al, 2007). Despite great research interest, factors that could contribute to scene memorability are not fully understood.…”

“…Particularly, fixation count has been repeatedly demonstrated to be associated with scene memory. For example, an increased fixation count during encoding is associated with better recognition on a trial-by-trial basis for scenes (Choe et al, 2017) and objects (Tatler & Tatler, 2013), suggesting that trial-level fixation count signals viewers' elaborate inspection of a scene and that elaborate inspection can enhance scene encoding (Winograd, 1981). Moreover, more preferred scenes produce more fixations and are better remembered later than less preferred scenes (Loftus, 1972), suggesting that population-level fixation count (i.e., the averaged fixation count across viewers) can be a proxy for an intrinsic property of a scene, such as interestingness (e.g., the more interesting a scene is, the more elaborate inspection viewers do).…”

“…Eye-tracking also enables the investigation of the relationship between where viewers look in scenes, i.e., fixation maps (Henderson, 2003;Pomplun et al, 1996;Wooding, 2002), and how scene memory is formed (Choe et al, 2017;Hollingworth, 2012;Olejarczyk et al, 2014;Ramey et al, in press;Tatler & Tatler, 2013). For example, the fixation map from a scene during intentional memorization is different from that during visual search (Castelhano et al, 2009), and the degree of difference in the fixation maps during memorization vs. visual search in the same scene could explain how visual search impaired incidental scene memory on a trial-by-trial basis (Choe et al, 2017).…”

“…Eye-tracking also enables the investigation of the relationship between where viewers look in scenes, i.e., fixation maps (Henderson, 2003;Pomplun et al, 1996;Wooding, 2002), and how scene memory is formed (Choe et al, 2017;Hollingworth, 2012;Olejarczyk et al, 2014;Ramey et al, in press;Tatler & Tatler, 2013). For example, the fixation map from a scene during intentional memorization is different from that during visual search (Castelhano et al, 2009), and the degree of difference in the fixation maps during memorization vs. visual search in the same scene could explain how visual search impaired incidental scene memory on a trial-by-trial basis (Choe et al, 2017). Similar to these approaches, one can also examine the consistency of fixation maps across viewers (also called inter-observer congruency or inter-subject consistency), i.e., fixation map consistency (Dorr et al, 2010;Torralba et al, 2006), which is a scene-specific, population-level measure (i.e., averaged over a group of participants for each scene) and often used in evaluating computational fixation prediction models by providing an upper bound of the performance that those models can achieve (Wilming et al, 2011).…”