Shared neural underpinnings of multisensory integration and trial-by-trial perceptual recalibration

Abstract

9Multisensory stimuli create behavioral flexibility, e.g. by allowing us to derive a weighted 10 combination of the information received by different senses. They also allow perception to 11 adapt to discrepancies in the sensory world, e.g. by biasing the judgement of unisensory cues 12 based on preceding multisensory evidence. While both facets of multisensory perception are 13 central for behavior, it remains unknown whether they arise from a common neural substrate. 14 In fact, very little is known about the neural mechanisms underlying multisensory perceptual 15 recalibration. To reveal these, we measured whole-brain activity using MEG while human 16 participants performed an audio-visual ventriloquist paradigm designed to reveal multisensory 17 integration within a trial, and the (trial-by-trial) recalibration of subsequent unisensory 18 judgements. Using single trial classification and behavioral modelling, we localized the 19 encoding of sensory information within and between trials, and determined the behavioral 20 relevance of candidate neural representations. While we found neural signatures of perceptual 21 integration within temporal and parietal regions, of these, only medial superior parietal activity 22 retained multisensory information between trials and combined this with current evidence to 23 mediate perceptual recalibration. These results suggest a common neural substrate of sensory 24 integration and trial-by-trial perceptual recalibration, and expose the medial superior parietal 25 cortex as a flexible hub that links present and previous evidence within and between senses 26 to guide behavior. 27Bayesian causal inference (30)(31)(32)(33)(34). Given that parietal regions also contribute to the 56 maintenance of sensory information within or between trials (35-40) this raises the possibility 57 that parietal regions are in fact mediating both the combination of sensory information within a 58 trial, and the influence of such an integrated representation on guiding subsequent adaptive 59 behavior. 60To link the neural mechanisms underlying multisensory integration and trial-by-trial 61 recalibration, we measured whole-brain activity using magnetoencephalography (MEG) while 62 human participants performed a spatial localization task. The paradigm was designed to reveal 63 the behavioral correlates of audio-visual integration (i.e. the ventriloquist effect, VE) and the 64 influence of this on the localization of a subsequent unisensory sounds (the ventriloquist 65 aftereffect, VAE) (6). Using single-trial classification we determined the relevant neural 66representations of auditory and visual spatial information and quantified when and where these 67 are influenced by previous sensory evidence. We then modelled the influence of these 68 candidate neural representations on the participant-specific trial-by-trial response biases. As 69 expected based on previous work, our results reveal neural origins of sensory integration in 70 superior temporal and parietal regions. Importantly, of these, only ...

Value network engagement and effects of memory-related processing during encoding and retrieval of value

“…While our task did include monetary incentives designed to elicit self-relevant value responses, it did not include trial-wise choices or reinforcements, which may have reduced the immediate self-relevance of our stimuli compared with those in previous studies. Beyond selfrelevant values, the observation of robust value-related activation in the superior parietal lobule and larger fronto-parietal network is in line with previous connections to choice action planning, trial-by-trial updating, and integration of bottom-up sensory information with top-down task goals (Andersen & Buneo, 2002;Cohen et al, 2014;Cole, Repovs, & Anticevic, 2014;Park & Kayser, 2019;Ptak, 2011;Tosoni et al, 2014;Yeo et al, 2011). Furthermore, fitting with value encoding in our task, value comparisons have been associated with superior parietal lobule engagement when value choices are made under restricted time limits, with the ventromedial prefrontal cortex keeping track of overall value (Domenech, Redouté, Koechlin, & Dreher, 2018;Jocham et al, 2014).…”

“…While select subregions of the posterior parietal cortex may be recruited to a greater extent during the value retrieval phase in our task, our results support the conclusion that posterior parietal regions are robustly associated with value rating behavior whether value-based information is being encoded or retrieved from memory. Recruitment of posterior parietal regions across value encoding and retrieval is consistent with research showing these regions support the accumulation of choice-relevant information and the transformation of integrated representations into behavioral responses (Andersen & Buneo, 2002;Clithero, Carter, & Huettel, 2009;Domenech et al, 2018;Kable & Glimcher, 2009;Park & Kayser, 2019). Further, superior subregions of the posterior parietal cortex have been associated with goal-directed valuation beyond bottom-up salience-driven responses (Kahnt et al, 2014;Koscik et al, 2020;Uncapher & Wagner, 2009).…”