Visual cortex signals a mismatch between regularity of auditory and visual streams

Andric, Michael; Davis, Ben; Hasson, Uri

doi:10.1016/j.neuroimage.2017.05.028

Cited by 8 publications

(5 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Areas encoding audiovisual entropy were largely non-overlapping with areas encoding entropy in unisensory visual and auditory series, which is consistent with prior work (Andric, Davis, & Hasson, 2017). This relatively modest overlap is also consistent with behavioral work suggesting that multisensory regularities are learned independently of regularities conveyed via their unisensory constituents (Seitz, Kim, van Wassenhove, & Shams, 2007).…”

Section: Cross-modal and Non-monotonic Sensitivity To Uncertaintysupporting

confidence: 86%

“…In general, we expected different brain systems to track uncertainty within auditory and visual streams, consistent with emerging views that different neural systems encode sequential structure or environmental regularities in different modalities (for recent reviews; see Armstrong, Siegelman, & Christiansen, 2015; Dehaene, Meyniel, Wacongne, Wang, & Pallier, 2015; Frost et al, 2015;Hasson, 2017;Milne, Wilson, & Christiansen, 2018). We further expected that the systems implicated in tracking the level of uncertainty in audiovisual stimuli would diverge from those tracking uncertainty for unisensory stimuli, as our recent work (Andric, Davis, & Hasson, 2017) indicates that audiovisual inputs trigger unique computations related to uncertainty.…”

Section: Introductionsupporting

confidence: 80%

See 1 more Smart Citation

Cross-modal and non-monotonic representations of statistical regularity are encoded in local neural response patterns

2018

Self Cite

View full text Add to dashboard Cite

Current neurobiological models assign a central role to predictive processes calibrated to environmental statistics. Neuroimaging studies examining the encoding of stimulus uncertainty have relied almost exclusively on manipulations in which stimuli were presented in a single sensory modality, and further assumed that neural responses vary monotonically with uncertainty. This has left a gap in theoretical development with respect to two core issues: (i) are there cross-modal brain systems that encode input uncertainty in way that generalizes across sensory modalities, and (ii) are there brain systems that track input uncertainty in a non-monotonic fashion? We used multivariate pattern analysis to address these two issues using auditory, visual and audiovisual inputs. We found signatures of cross-modal encoding in frontoparietal, orbitofrontal, and association cortices using a searchlight cross-classification analysis where classifiers trained to discriminate levels of uncertainty in one modality were tested in another modality. Additionally, we found widespread systems encoding uncertainty non-monotonically using classifiers trained to discriminate intermediate levels of uncertainty from both the highest and lowest uncertainty levels. These findings comprise the first comprehensive report of cross-modal and non-monotonic neural sensitivity to statistical regularities in the environment, and suggest that conventional paradigms testing for monotonic responses to uncertainty in a single sensory modality may have limited generalizability.

show abstract

Section: Cross-modal and Non-monotonic Sensitivity To Uncertaintysupporting

confidence: 86%

Section: Introductionsupporting

confidence: 80%

Cross-modal and non-monotonic representations of statistical regularity are encoded in local neural response patterns

2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…While studies directly investigating mismatch signals in response to multimodal sensory inputs are rare, previous research indicates a ubiquitous role for cross‐modal probabilistic learning. The brain tends to automatically integrate auditory, somatosensory, and visual stimuli during sequence processing (Bresciani et al, 2006 , 2008 ; Frost et al, 2015 ) and cross‐modal perceptual associations can influence statistical learning of sequence regularities (Andric et al, 2017 ; Parmentier et al, 2011 ), modulate MMRs (Besle et al, 2005 ; Butler et al, 2012 ; Friedel et al, 2020 ; Kiat, 2018 ; Zhao et al, 2015 ) and influence subsequent unimodal processing in various ways (Shams et al, 2011 ). Recent advances in modeling Bayesian causal inference suggest that the main computational stages of multimodal inference evolve along a multisensory hierarchy involving early sensory segregation followed by mid‐latency sensory fusion and late Bayesian causal inference (Cao et al, 2019 ; Rohe et al, 2019 ; Rohe & Noppeney, 2015 ).…”

Section: Introductionmentioning

confidence: 99%

EEG mismatch responses in a multimodal roving stimulus paradigm provide evidence for probabilistic inference across audition, somatosensation, and vision

Grundei

Schröder

Gijsen

et al. 2023

Human Brain Mapping

View full text Add to dashboard Cite

The human brain is constantly subjected to a multimodal stream of probabilistic sensory inputs. Electroencephalography (EEG) signatures, such as the mismatch negativity (MMN) and the P3, can give valuable insight into neuronal probabilistic inference. Although reported for different modalities, mismatch responses have largely been studied in isolation, with a strong focus on the auditory MMN. To investigate the extent to which early and late mismatch responses across modalities represent comparable signatures of uni‐ and cross‐modal probabilistic inference in the hierarchically structured cortex, we recorded EEG from 32 participants undergoing a novel tri‐modal roving stimulus paradigm. The employed sequences consisted of high and low intensity stimuli in the auditory, somatosensory and visual modalities and were governed by unimodal transition probabilities and cross‐modal conditional dependencies. We found modality specific signatures of MMN (~100–200 ms) in all three modalities, which were source localized to the respective sensory cortices and shared right lateralized prefrontal sources. Additionally, we identified a cross‐modal signature of mismatch processing in the P3a time range (~300–350 ms), for which a common network with frontal dominance was found. Across modalities, the mismatch responses showed highly comparable parametric effects of stimulus train length, which were driven by standard and deviant response modulations in opposite directions. Strikingly, P3a responses across modalities were increased for mispredicted stimuli with low cross‐modal conditional probability, suggesting sensitivity to multimodal (global) predictive sequence properties. Finally, model comparisons indicated that the observed single trial dynamics were best captured by Bayesian learning models tracking unimodal stimulus transitions as well as cross‐modal conditional dependencies.

show abstract

“…19 The visual cortex is a junction for integration of temporally-extended auditory and visual inputs. 20 Retinopathy-induced impaired vision is suggested to be the initial factor leading to the visual functional network change and the disorder of visual and auditory function. The FCS was decreased in DR patients among Heschl's gyri, bilateral fusiform gyri, amygdala, left superior parietal gyrus, and left parahippocampal.…”

Section: Discussionmentioning

confidence: 99%

<p>Resting-State Functional MRI Study: Connection Strength of Brain Networks in DR Patients</p>

Dai

et al. 2019

NDT

View full text Add to dashboard Cite

Purpose: To explore the functional connection strength (FCS) changes of brain networks in diabetic retinopathy (DR) patients and uncover the underlying mechanism. Methods and Materials: Twenty-one patients with DR and 21 age-and sex-matched healthy controls were enrolled from August 2012 to September 2014. Subjects were scanned using 3T MR with blood-oxygen-level dependent (BOLD) and 3-dimension fast spoiled gradient echo (3D-FSPGR) sequences. MR data was analyzed via preprocessing and functional network construction. After a group comparison, components of brain networks with significant group differences were extracted and the FCS of the brain network was evaluated. The brain areas were compared between patients and controls. P-values less than 0.05 were considered statistically significant. Connection strength was evaluated with alphasim, P<0.01. Results: The component maps of altered brain networks with quantified FCS were obtained in DR patients, demonstrating more disconnections mainly in the bilateral Heschl's gyrus, left cuneus, left occipital lobe, bilateral amygdala, left parahippocampal, bilateral fusiform, and left superior parietal in the patients group compared to the healthy controls (P<0.01), while compensations may occur in the frontal-cingulum region, as well as among the right caudate, left thalamus, left inferior temporal lobe, and middle orbital frontal lobe. Conclusion: Brain network connections, decreased in the brain areas of which in charging with cognition and visual function, suggests that DR patients might have cognitive decline and visual function loss. However, there might be a frontal compensatory circle in patients with DR.

show abstract

Visual cortex signals a mismatch between regularity of auditory and visual streams

Cited by 8 publications

References 52 publications

Cross-modal and non-monotonic representations of statistical regularity are encoded in local neural response patterns

Cross-modal and non-monotonic representations of statistical regularity are encoded in local neural response patterns

EEG mismatch responses in a multimodal roving stimulus paradigm provide evidence for probabilistic inference across audition, somatosensation, and vision

<p>Resting-State Functional MRI Study: Connection Strength of Brain Networks in DR Patients</p>

Contact Info

Product

Resources

About