Neural correlates of multisensory reliability and perceptual weights emerge at early latencies during audio‐visual integration

Abstract

To make accurate perceptual estimates, observers must take the reliability of sensory information into account. Despite many behavioural studies showing that subjects weight individual sensory cues in proportion to their reliabilities, it is still unclear when during a trial neuronal responses are modulated by the reliability of sensory information or when they reflect the perceptual weights attributed to each sensory input. We investigated these questions using a combination of psychophysics, EEG-based neuroimaging and single-trial decoding. Our results show that the weighted integration of sensory information in the brain is a dynamic process; effects of sensory reliability on task-relevant EEG components were evident 84 ms after stimulus onset, while neural correlates of perceptual weights emerged 120 ms after stimulus onset. These neural processes had different underlying sources, arising from sensory and parietal regions, respectively. Together these results reveal the temporal dynamics of perceptual and neural audio-visual integration and support the notion of temporally early and functionally specific multisensory processes in the brain.

Consistent pre-stimulus influences on auditory perception across the lifespan

“…To link pre-stimulus activity with perception we used the same statistical modelling approach as in our previous study (Kayser et al., 2016). We computed pre-stimulus activity in task-relevant EEG components extracted using multivariate linear discriminant analysis (Boyle et al., 2017; Kayser et al., 2016; Parra et al., 2005; Philiastides, 2006; Ratcliff et al., 2009). We searched for discriminant components within the EEG data that best discriminated between the frequency conditions (i.e.…”

Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions

“…Therefore, this reaffirms our assumption that associative congruency shapes multisensory decision formation, thus improving either or both the speed and accuracy of choice. Similarly, multisensory enhancements during perceptual decision formation have been found with interactions occurring in neural signals at very short latencies ( Boyle et al., 2017 ; Cappe et al., 2010 ; Foxe et al., 2000 ; 2002 ; Foxe and Schroeder, 2005 ; Molholm et al., 2002 ; 2006 ; Sperdin et al., 2009 ). Importantly, in our study the early modulation we observed suggests such enhancements are not exclusively multisensory, since on each trial only a single sensory stimulus was presented.…”

Neural dynamics of cue reliability in perceptual decisions

“…This way we revealed latencies and electrodes where VEP amplitudes increase with cue reliability at the same rate that behavioral discriminability increases with reliability. Previous work studied how information about uncertain sensory evidence is neurally represented and by linking this representation to behavioral outcomes they suggested a unified psychophysical-neural model (Boyle et al, 2017;Philiastides & Sajda, 2006). The main advantage of our methodology lies in establishing an equivalence with the sensitivity to cue reliability (i.e.…”