Multi-step planning of eye movements in visual search

Abstract

The capability of directing gaze to relevant parts in the environment is crucial for our survival. Computational models have proposed quantitative accounts of human gaze selection in a range of visual search tasks. Initially, models suggested that gaze is directed to the locations in a visual scene at which some criterion such as the probability of target location, the reduction of uncertainty or the maximization of reward appear to be maximal. But subsequent studies established, that in some tasks humans instead direct their gaze to locations, such that after the single next look the criterion is expected to become maximal. However, in tasks going beyond a single action, the entire action sequence may determine future rewards thereby necessitating planning beyond a single next gaze shift. While previous empirical studies have suggested that human gaze sequences are planned, quantitative evidence for whether the human visual system is capable of finding optimal eye movement sequences according to probabilistic planning is missing. Here we employ a series of computational models to investigate whether humans are capable of looking ahead more than the next single eye movement. We found clear evidence that subjects’ behavior was better explained by the model of a planning observer compared to a myopic, greedy observer, which selects only a single saccade at a time. In particular, the location of our subjects’ first fixation differed depending on the stimulus and the time available for the search, which was well predicted quantitatively by a probabilistic planning model. Overall, our results are the first evidence that the human visual system’s gaze selection agrees with optimal planning under uncertainty.

Tuning movement for sensing in an uncertain world

“…Hence, the opportunity to visit another location in space that is statistically independent—yet contains a similar amount of predicted information—gives an animal an opportunity to mitigate unmodeled uncertainties through the expenditure of energy for movement. This is supported by experiments in human visual search suggesting that saccades are planned in a multi-stage manner for coverage of information towards the task-relevant goal rather than aiming for information maximization ( Yang et al, 2016 ; Hoppe and Rothkopf, 2019 ). For example, a model to predict human visual scan paths found 70% of the measured fixation locations were efficient from an information maximization perspective, but there were many fixations (≈30%) that were not purely for maximizing information and attributed in part to perceptual or motor noise ( Yang et al, 2016 ).…”

“…For example, weakly electric fish will track and stay near a moving refuge, but in addition to the large motions needed to stay near the refuge, there are small whole-body oscillations—an electrosensory analog to microsaccades ( Video 1 and Figure 1—figure supplement 1 ; Stamper et al, 2012 ). Similarly, in behaviors where animals sample discretely over time, animals vary their sampling frequency or the location at which samples are taken, as observed in bats, rats, beaked whales, humans, and pulse electric fish ( Yovel et al, 2010 ; Mitchinson et al, 2007 ; Hartmann, 2001 ; Kothari et al, 2018 ; Caputi et al, 2003 ; Pluta and Kawasaki, 2008 ; Nelson and MacIver, 2006 ; Schnitzler et al, 2003 ; Madsen et al, 2005 ; Yang et al, 2016 ; Hoppe and Rothkopf, 2019 ).…”

Learning in the eyes: specific changes in gaze patterns track explicit and implicit visual learning

“…Studies in the first group focused on the interaction between explicitly or implicitly defined but already available internal knowledge and eye movements in various tasks by investigating the number and position of fixations necessary for finding a target in a display or making a decision (Chukoskie et al, 2013;Hoppe & Rothkopf, 2019;Morvan & Maloney, 2012;Najemnik & Geisler, 2005;Peterson & Kramer, 2001;Yang et al, 2017). Studies in the other group investigated the effect of learning on eye movements but only in terms of learning temporal regularities and adjusting the timing of fixations accordingly (Glimcher, 2003;Hoppe & Rothkopf, 2016).…”

Intuitive physical reasoning about objects’ masses transfers to a visuomotor decision task consistent with Newtonian physics

“…The present model used a cost function to account for the costs and benefits implicit in our participants visuomotor behavior and may encompass external and internal cost related to different task components, perceptual, cognitive, biomechanical costs and preferences. Inferring such costs and benefits has been shown to be crucial for the understanding of visuomotor behavior [34][35][36].…”