Multisensory Integration Uses a Real-Time Unisensory–Multisensory Transform

Abstract

The manner in which the brain integrates different sensory inputs to facilitate perception and behavior has been the subject of numerous speculations. By examining multisensory neurons in cat superior colliculus, the present study demonstrated that two operational principles are sufficient to understand how this remarkable result is achieved: (1) unisensory signals are integrated continuously and in real time as soon as they arrive at their common target neuron and (2) the resultant multisensory computation is modified in shape and timing by a delayed, calibrating inhibition. These principles were tested for descriptive sufficiency by embedding them in a neurocomputational model and using it to predict a neuron's moment-by-moment multisensory response given only knowledge of its responses to the individual modality-specific component cues. The predictions proved to be highly accurate, reliable, and unbiased and were, in most cases, not statistically distinguishable from the neuron's actual instantaneous multisensory response at any phase throughout its entire duration. The model was also able to explain why different multisensory products are often observed in different neurons at different time points, as well as the higher-order properties of multisensory integration, such as the dependency of multisensory products on the temporal alignment of crossmodal cues. These observations not only reveal this fundamental integrative operation, but also identify quantitatively the multisensory transform used by each neuron. As a result, they provide a means of comparing the integrative profiles among neurons and evaluating how they are affected by changes in intrinsic or extrinsic factors. Multisensory integration is the process by which the brain combines information from multiple sensory sources (e.g., vision and audition) to maximize an organism's ability to identify and respond to environmental stimuli. The actual transformative process by which the neural products of multisensory integration are achieved is poorly understood. By focusing on the millisecond-by-millisecond differences between a neuron's unisensory component responses and its integrated multisensory response, it was found that this multisensory transform can be described by two basic principles: unisensory information is integrated in real time and the multisensory response is shaped by calibrating inhibition. It is now possible to use these principles to predict a neuron's multisensory response accurately armed only with knowledge of its unisensory responses.

Formal models and quantitative measures of multisensory integration: a selective overview

“…Similarly, any crossmodal response larger than the largest unisensory response but smaller than the sum might be "misinterpreted" as response depression (Stein et al, 2009). Nevertheless, the additive version is often used, in addition to the original CRE, to measure the amount of "superadditivity" (e.g., Miller et al, 2017). At this point, it is important to point out that our discussion here is necessarily incomplete given that the important case of multisensory inhibition has been omitted, due to limits of space.…”

“…They go beyond models like the one by Ohshiro et al (2011) (i) in representing the temporal profile of multisensory enhancement or suppression (Rowland et al, 2007), (ii) in explicitly taking into account the finding that multisensory integration in the SC is mediated by descending inputs from association cortex (Jiang et al, 2001), and (iii) in addressing questions about the emergence and maturation of multisensory integration in dependence of the environment . In a recent computational model, the continuous-time multisensory model (CTMM) by Miller et al (2017), the authors show that real-time integration and delayed, calibrating inhibition are sufficient to predict the individual moment-by-moment multisensory response given only knowledge of the associated unisensory responses. Some aspects of CTMM are reminiscent of the MCD correlation detector by Parise & Ernst (2016), and it might be of interest to investigate this further.…”

Electrophysiological correlates of emotional crossmodal processing in binge drinking

“…In line with the first analyses, the comparisons based on subtraction waveforms also indicated that cerebral changes during crossmodal processing in BDs seem related to later stages. According to previous studies (Joassin et al, 2004;Miller, Stein, & Rowland, 2017), crossmodal integration is first related to complex visuo-auditory interactions at perceptual stages, followed by the building of an integrated crossmodal representation at later stages, together with facilitatory or inhibitory processes. Nevertheless, although results appear to be consistent with the literature and throughout our different analysis steps, findings also reveal that BDs process faster than MDs do on the last component of the occipital site, although no difference was found with NDs.…”

A Combinatorial Premotor Neural Code: Transformation Of Sensory Information Into Meaningful Rhythmic Motor Output By A Network Of Heterogeneous Modulatory Neurons

“…To encode a unique condition, network responses have to be distinct from the unimodal responses. Super-and sub-additive responses to bimodal input are quite common features in individual neurons, for example in integrating and non-integrating interneurons of the superior colliculus (Miller et al, 2017). This contrasts with some studies, such as moth tracking behavior (Roth et al, 2016), in which multimodal responses are the linear sum of two unimodal inputs.…”

“…information in midbrain and brainstem: auditory and visual information yield disparate firing in the superior colliculus (Meredith et al, 1987;Miller et al, 2017), and neurons in the solitary nucleus are suggested to use rate coding or the specific timing of spikes relative to one another (often labeled temporal coding) for taste and odorant discrimination (Escanilla et al, 2015). A common argument for rate coding is that it may increase multimodal information capacity, i.e.…”