Behavioral and oculomotor evidence for visual simulation of object movement

Abstract

We regularly interact with moving objects in our environment. Yet, little is known about how we extrapolate the future movements of visually perceived objects. One possibility is that movements are experienced by a mental visual simulation, allowing one to internally picture an object's upcoming motion trajectory, even as the object itself remains stationary. Here we examined this possibility by asking human participants to make judgments about the future position of a falling ball on an obstacle-filled display. We found that properties of the ball's trajectory were highly predictive of subjects' reaction times and accuracy on the task. We also found that the eye movements subjects made while attempting to ascertain where the ball might fall had significant spatiotemporal overlap with those made while actually perceiving the ball fall. These findings suggest that subjects simulated the ball's trajectory to inform their responses. Finally, we trained a convolutional neural network to see whether this problem could be solved by simple image analysis as opposed to the more intricate simulation strategy we propose. We found that while the network was able to solve our task, the model's output did not effectively or consistently predict human behavior. This implies that subjects employed a different strategy for solving our task, and bolsters the conclusion that they were engaging in visual simulation. The current study thus provides support for visual simulation of motion as a means of understanding complex visual scenes and paves the way for future investigations of this phenomenon at a neural level.

A role for visual areas in physics simulations

“…Further, we showed that participants' reaction times were greater and accuracy was lower on high simulation uncertainty boards relative to low simulation uncertainty boards. (Ahuja & Sheinberg, 2019). Here, we sought to replicate these effects from our past experiments to verify that participants' behavior on the task matched what we have previously presented as evidence in favor of simulation.…”

“…An example of a low simulation uncertainty board is shown in Figure 2C. A dynamic demonstration of these uncertainty categories is shown in a supplementary video from our previous paper (Ahuja & Sheinberg, 2019). Simulation length and simulation uncertainty categories were counterbalanced, leading to six trials per length and uncertainty combination per block (2 length categories X 2 uncertainty categories X 6 = 24 trials).…”

“…In a previous study, we designed a novel task in which participants had to predict a ball's likely trajectory as it fell through an obstacle filled display (Ahuja & Sheinberg, 2019). We refer to this as the ball fall task.…”

A Role for Visual Areas in Physics Simulations

“…As seen in this figure, participants were generally very good at all three task variants and performed far better than chance, which was defined as 50% correct (Simulation: 89.8%, [t 11 = 39.745, p < 0.001]; Perception: 99.6% [t 11 = 199.23, p < 0.001]; Control: 99.4% [t 11 = 155.45, p < 0.001]). Previously, we showed that people perform the ball fall task via simulation by relating their reaction times and accuracy to two simulation-based metrics – length and uncertainty (Ahuja & Sheinberg, 2019). Specifically, we showed that as simulation length increased, so did participants’ reaction times.…”

Looking into the past: Eye-tracking mental simulation in physical inference

“…Eye-tracking in particular has proven a promising approach. In a variety of intuitive physical tasks, researchers have captured human eye-data to investigate claims about mental simulation (Ahuja & Sheinberg, 2019;Crespi, Robino, Silva, & de'Sperati, 2012;Gerstenberg, Peterson, Goodman, Lagnado, & Tenenbaum, 2017). Eye-data yield a moment-to-moment trace of human behavior throughout the process of making a physical judgment, augmenting standard behavioral measures and providing rich empirical fodder for making inferences about human cognition.…”