Sensory Evidence Accumulation Using Optic Flow in a Naturalistic Navigation Task

Abstract

Sensory evidence accumulation is considered a hallmark of decision-making in noisy environments. Integration of sensory inputs has been traditionally studied using passive stimuli, segregating perception from action. Lessons learned from this approach, however, may not generalize to ethological behaviors like navigation, where there is an active interplay between perception and action. We designed a sensory-based sequential decision task in virtual reality in which humans and monkeys navigated to a memorized location by integrating optic flow generated by their own joystick movements. A major challenge in such closed-loop tasks is that subjects' actions will determine future sensory input, causing ambiguity about whether they rely on sensory input rather than expectations based solely on a learned model of the dynamics. To test whether subjects integrated optic flow over time, we used three independent experimental manipulations, unpredictable optic flow perturbations, which pushed subjects off their trajectory; gain manipulation of the joystick controller, which changed the consequences of actions; and manipulation of the optic flow density, which changed the information borne by sensory evidence. Our results suggest that both macaques (male) and humans (female/male) relied heavily on optic flow, thereby demonstrating a critical role for sensory evidence accumulation during naturalistic action-perception closed-loop tasks. SIGNIFICANCE STATEMENT The temporal integration of evidence is a fundamental component of mammalian intelligence. Yet, it has traditionally been studied using experimental paradigms that fail to capture the closed-loop interaction between actions and sensations inherent in real-world continuous behaviors. These conventional paradigms use binary decision tasks and passive stimuli with statistics that remain stationary over time. Instead, we developed a naturalistic visuomotor visual navigation paradigm that mimics the causal structure of real-world sensorimotor interactions and probed the extent to which participants integrate sensory evidence by adding task manipulations that reveal complementary aspects of the computation.

Inductive biases of neural specialization in spatial navigation

“…Together, these findings shed light on how to bridge the gap between AI and neuroscience [3, 4, 18, 19]: from AI to neuroscience, our work validates the rationale of the brain’s inductive biases in spatial navigation [20, 17]. From neuroscience to AI, our work exemplifies how insights from the brain can inspire the development of AI with better generalization abilities.…”

“…Training agents with greater uncertainty in observation than prediction biases agents toward ignoring observations in belief formation, which then cripples their generalization abilities in novel tasks that require the use of novel observations to construct the belief, instead of the outdated internal model. Therefore, to generalize, agents must develop prior knowledge of relying more on observation, similar to macaques and humans in this task [17].…”

Predictive steering: Integration of artificial motor signals in self-motion estimation

“…Our study is one among the few recent studies that tested self-motion perception under a direct coupling between the actions of the participants and the sensory feedback (25)(26)(27). These recent experiments imposed fewer artificial constraints than the traditional open-loop psychophysical paradigms on self-motion perception (e.g., [37][38][39][40][41].…”

“…Under the assumption that the firing rates of neurons in the VN reflect sensory predictions errors (16, 45), this suggests that no steering-related predictions about the vestibular reafference are made in the VN. Even though the vestibular cerebellum is often suggested to house the internal model for self-motion estimation because of its projections to the vestibular nuclei (46), the internal model of the mapping between steering movements and self-motion seems located on a more downstream level within the vestibular processing pathway (25). This is in line with observations during the processing of the visual reafference of steering movements; Page and Duffy (22) reported that neurons in the medial superior temporal area in monkeys responded differently to optic flow cues resulting from steering movements compared to passive viewing of the same optic flow cues.…”

“…In the present study we address this issue in humans, testing whether and how steering-related signals are used in self-motion perception. Recent virtual reality experiments examined how humans (and monkeys) virtually navigated to a memorized location by integrating optic flow generated by their own joystick movements (25)(26)(27). Biases in their steering depended on optic flow density, as a marker of the reliability of sensory evidence, and the control gain of the joystick, as a measure of the internal model prediction of the optic flow, suggesting that the brain combined both signals in the percept of non-vestibular self-motion.…”