Segregation and Integration of Cortical Information Processing Underlying Cross-Modal Perception

Abstract

Visual cues from the speaker’s face influence the perception of speech. An example of this influence is demonstrated by the McGurk-effect where illusory (cross-modal) sounds are perceived following presentation of incongruent audio–visual (AV) stimuli. Previous studies report the engagement of specific cortical modules that are spatially distributed during cross-modal perception. However, the limits of the underlying representational space and the cortical network mechanisms remain unclear. In this combined psychophysical and electroencephalography (EEG) study, the participants reported their perception while listening to a set of synchronous and asynchronous incongruent AV stimuli. We identified the neural representation of subjective cross-modal perception at different organizational levels — at specific locations in sensor space and at the level of the large-scale brain network estimated from between-sensor interactions. We identified an enhanced positivity in the event-related potential peak around 300 ms following stimulus onset associated with cross-modal perception. At the spectral level, cross-modal perception involved an overall decrease in power at the frontal and temporal regions at multiple frequency bands and at all AV lags, along with an increased power at the occipital scalp region for synchronous AV stimuli. At the level of large-scale neuronal networks, enhanced functional connectivity at the gamma band involving frontal regions serves as a marker of AV integration. Thus, we report in one single study that segregation of information processing at individual brain locations and integration of information over candidate brain networks underlie multisensory speech perception.

Biophysical mechanisms governing large‐scale brain network dynamics underlying individual‐specific variability of perception

“…Robust oscillations observed from macroscopic recordings such as EEG/MEG are an outcome of network interactions among local subpopulations of excitatory and inhibitory neurons (Becker, Knock, Ritter, & Jirsa, 2015; Deco, Rolls, & Romo, 2010; Wilson & Cowan, 1972). Empirically, such interactions result in global coherence dynamics (Kumar et al., 2017). Our study demonstrates how distinct global coherence patterns correlate with inter‐individual heterogeneity.…”

“…Traditionally, studies have focused on identifying the neural correlates in terms of candidate brain regions or interactions among specific regions of interest that are responsible for the observed heterogeneity (Nath & Beauchamp, 2012). A more emerging understanding, however, suggests the existence of large-scale networks of brain regions facilitating perceptual processing (Beauchamp, 2015;Keil et al, 2012;Kumar et al, 2017). Neuronal oscillations have been identified as a key marker of neuronal information processing in the relevant brain regions that needs to be fully explored to shape an individual's perceptual experience (Keil et al, 2012;Keil & Senkowski, 2018;Kumar et al, 2016Kumar et al, , 2017.…”

“…A more emerging understanding, however, suggests the existence of large-scale networks of brain regions facilitating perceptual processing (Beauchamp, 2015;Keil et al, 2012;Kumar et al, 2017). Neuronal oscillations have been identified as a key marker of neuronal information processing in the relevant brain regions that needs to be fully explored to shape an individual's perceptual experience (Keil et al, 2012;Keil & Senkowski, 2018;Kumar et al, 2016Kumar et al, , 2017. Robust oscillations observed from macroscopic recordings such as EEG/MEG are an outcome of network interactions among local subpopulations of excitatory and inhibitory neurons (Becker, Knock, Ritter, & Jirsa, 2015;Deco, Rolls, & Romo, 2010;Wilson & Cowan, 1972).…”

“…Recent studies have indicated that beyond a specific region of interest, a large‐scale network of oscillatory brain networks underwrite the McGurk effect (Kumar et al., 2016; Kumar, Kumar, Roy, & Banerjee, 2017). A key question emerges, how robust is this network across individuals and whether the organization of such networks is contingent on the stimulus configurations.…”

The integration of facial expression and vocal emotion and its brain mechanism

“…也可整合成为一致情绪客体, 如 AV > [unimodal auditory (A) + unimodal visual (V)], 或理解为 1 + 1 > 2; 再次, 遵循颠倒效应规则(the principal of inverse effectiveness), 参与整合的单通道传递信 息较为模糊, 或是双通道信息都较为模糊时, 整 合效 应 更 为显 著 (Stein & Stanford, 2008;Stein, Stanford, Ramachandran, Perrault, & Rowland, 2009 (Collignon et al, 2008;Focker, Gondan, & Roder, 2011;Zinchenko, Obermeier, Kanske, Schroger, & Kotz, 2017) (Calvo & Nummenmaa, 2016;Collignon et al, 2008;Focker et al, 2011;Klasen et al, 2012;Kumar et al, 2018;Muller et al, 2012…”