Neural correlates of reliability-based cue weighting during multisensory integration

Abstract

Integration of multiple sensory cues is essential for precise and accurate perception and behavioral performance, yet the reliability of sensory signals can vary across modalities and viewing conditions. Human observers typically employ the optimal strategy of weighting each cue in proportion to its reliability, but the neural basis of this computation remains poorly understood. We trained monkeys to perform a heading discrimination task from visual and vestibular cues, varying cue reliability at random. Monkeys appropriately placed greater weight on the more reliable cue, and population decoding of neural responses in area MSTd nicely predicted behavioral cue weighting, including modest deviations from optimality. We further show that the mathematical combination of visual and vestibular inputs by single neurons is generally consistent with recent theories of optimal probabilistic computation in neural circuits. These results provide direct evidence for a neural mechanism mediating a simple and widespread form of statistical inference.

How Optic Flow and Inertial Cues Improve Motion Perception

“…Indeed, empirically measured heading discrimination thresholds during combined vestibular -visual heading perception were close to the predictions produced by a statistically optimal cue combination rule (Gu et al 2008;Fetsch et al 2009Fetsch et al , 2011Fetsch et al , 2013Butler et al 2010Butler et al , 2015de Winkel et al 2013). …”

“…1). Fetsch et al (2011) further showed that congruent MSTd neurons also satisfy the second prediction of optimal cue integration theory (Eq. 2).…”

“…In particular, as illustrated by the psychometric functions in Figure 2C, the precision of perception in a heading discrimination task (Fig. 2B) increases when both vestibular (i.e., inertial) and visual (i.e., optic flow) signals indicate self-motion as compared with when either cue is present alone (Gu et al 2008;Fetsch et al 2009Fetsch et al , 2011Fetsch et al , 2013Butler et al 2010Butler et al , 2015de Winkel et al 2013). Precision is the sensitivity or threshold (also called "reliability," computed as the "sigma" of the cumulative Gaussian fit of a psychometric function) (C, vertical dashed lines).…”

“…In contrast, the neural basis of egocentric heading direction has been primarily investigated in macaques. This review provides a summary of what is currently known about how visual and vestibular cues influence and control our perceived egocentric heading direction (see also Britten 2008;Angelaki et al 2009Angelaki et al , 2011Fetsch et al 2010Fetsch et al , 2011DeAngelis and Angelaki 2012).…”

Sensorimotor basis of motor vehicle control

“…Fetsch, Turner et al (2009), by manipulating the reliability of visual and vestibular cues, showed that primates can re-weight each cue dynamically on a trial-by-trial basis, consistent with a Bayesian assumption that the most reliable cue receive more weight. Recent studies generally agree with this finding (Butler, Smith et al 2010, Campos, Byrne et al 2010, Fetsch, Pouget et al 2012, Saunders 2014, Butler, Campos et al 2015, except that de Winkel, Weesie et al…”