EEG mismatch responses in a multi-modal roving stimulus paradigm provide evidence for probabilistic inference across audition, somatosensation and vision

Abstract

The human brain is constantly subjected to a multi-modal stream of probabilistic sensory inputs. EEG signatures, such as the mismatch negativity (MMN) and the P3, can give valuable insight into neuronal probabilistic inference. Although reported for different modalities, mismatch responses have largely been studied in isolation, with a strong focus on the auditory MMN. To investigate the extent to which early and late mismatch responses across modalities represent comparable signatures of uni- and cross-modal probabilistic inference in the hierarchically structured cortex, we recorded EEG from 32 participants undergoing a novel tri-modal roving stimulus paradigm. The employed sequences consisted of high and low intensity stimuli in the auditory, somatosensory and visual modalities and were governed by uni-modal transition probabilities and cross-modal conditional dependencies. We found modality specific signatures of MMN (~100-200ms) in all three modalities, which were source localized to the respective sensory cortices and shared right lateralized pre-frontal sources. Additionally, we identified a cross-modal signature of mismatch processing in the P3a time range (~300-350ms), for which a common network with frontal dominance was found. Across modalities, the mismatch responses showed highly comparable parametric effects of stimulus train length, which were driven by standard and deviant response modulations in opposite directions. Strikingly, the P3a responses across modalities were increased for mispredicted compared to predicted and unpredictable stimuli, suggesting sensitivity to cross-modal predictive information. Finally, model comparisons indicated that the observed single trial dynamics were best captured by Bayesian learning models tracking uni-modal stimulus transitions as well as cross-modal conditional dependencies.

A multimodal cortical network of sensory expectation violation revealed by fMRI

“…An early finding, which has become a focus of the research on statistical sensory learning, is the modulation of the auditory MMN by standard repetition, that is, its amplitude increase with the number of prior standard presentations (Cowan et al, 1993; Haenschel et al, 2005; Imada et al, 1993; Sams et al, 1983). More recently, such modulation was shown beyond the auditory modality for both, MMN and P3 (Gijsen et al, 2021; Grundei et al, 2023). Although discussion persists about the contribution of stimulus specific adaptation in early sensory regions to these effects (Jääskelainen et al, 2004; May & Tiitinen, 2010), evidence converges to the view that increasing top‐down modulations in response to the repeating stimulus accounts best for this observation (Auksztulewicz et al, 2017; Auksztulewicz & Friston, 2016; Baldeweg, 2006; Ewbank et al, 2011; Garrido, Kilner, Stephan, & Friston, 2009; Langner et al, 2011; Summerfield et al, 2008; Todorovic & de Lange, 2012), particularly as studies have highlighted the dependence of MMRs on stimulus expectation based on statistical regularities as opposed to mere changes in stimulus properties (Bendixen et al, 2012; Heilbron & Chait, 2018; Wacongne et al, 2011).…”

“…While the earlier P3a sub‐component is task‐independent, drawing observers' attention to novel or unexpected stimuli (Escera et al, 2000; Friedman et al, 2001; Knight & Scabini, 1998), the later P3b response is more sensitive to task‐related target stimuli (Duncan et al, 2009; Polich, 2007). Although extensively researched in the auditory domain, the P3 is known for its modality independent characteristics (Linden, 2005; Polich, 2007) and has been equivalently reported for somatosensation (Andersen & Lundqvist, 2019; Gijsen et al, 2021; Ostwald et al, 2012; Shen et al, 2018a, 2018b; Yamaguchi & Knight, 1991, 1992) and vision (Conill, 1998; Duncan et al, 2009; Picton, 1992; Zhang et al, 2022), and has been described across senses in response to multi‐modal sequences (Grundei et al, 2023). The generating sources underlying the P3 are thought to be distributed in a fronto‐parietal network, involving inferior frontal, anterior cingulate and temporo‐parietal regions, with some indications for pronounced frontal contributions for the P3a and parietal dominance for the P3b (Linden, 2005; Polich, 2007).…”

“…The major neuronal generator of the auditory MMN is found in the auditory cortex where the exact location depends on the eliciting sound features and their complexity (Alho, 1995; Giard et al, 1990; Molholm et al, 2005; Sabri et al, 2004), with additional contributions from inferior frontal cortex (Deouell, 2007; Doeller et al, 2003; Molholm et al, 2005; Opitz et al, 2002; Rinne et al, 2005; Shalgi & Deouell, 2007). Similarly, the neuronal sources underlying the somatosensory MMN are found in primary and secondary somatosensory cortices (Akatsuka, Wasaka, Nakata, Kida, Hoshiyama, et al, 2007; Akatsuka, Wasaka, Nakata, Kida, & Kakigi, 2007; Andersen & Lundqvist, 2019; Butler et al, 2012; Gijsen et al, 2021; Grundei et al, 2023; Naeije et al, 2016; Ostwald et al, 2012; Spackman et al, 2010), in combination with inferior frontal cortex (Allen et al, 2016; Chen et al, 2008; Fardo et al, 2017; Grundei et al, 2023; Ostwald et al, 2012). The combination of sensory and frontal sources is also indicated to underlie the visual MMN (Hedge et al, 2015; Iglesias et al, 2013; Kimura et al, 2011; Pazo‐Alvarez et al, 2003; Yucel et al, 2007).…”

“…Although discussion persists about the contribution of stimulus specific adaptation in early sensory regions to these effects (Jääskelainen et al, 2004; May & Tiitinen, 2010), evidence converges to the view that increasing top‐down modulations in response to the repeating stimulus accounts best for this observation (Auksztulewicz et al, 2017; Auksztulewicz & Friston, 2016; Baldeweg, 2006; Ewbank et al, 2011; Garrido, Kilner, Stephan, & Friston, 2009; Langner et al, 2011; Summerfield et al, 2008; Todorovic & de Lange, 2012), particularly as studies have highlighted the dependence of MMRs on stimulus expectation based on statistical regularities as opposed to mere changes in stimulus properties (Bendixen et al, 2012; Heilbron & Chait, 2018; Wacongne et al, 2011). Computational modeling of EEG dynamics has indicated that probabilistic learning of environmental statistics underlies MMRs in the auditory (Lecaignard et al, 2022; Lieder, Daunizeau, et al, 2013; Weber et al, 2020), somatosensory (Gijsen et al, 2021; Grundei et al, 2023; Ostwald et al, 2012) and visual modality (Stefanics et al, 2018). This view is supported by studies showing deviant responses to abstract rule violations (Näätänen et al, 2010; Paavilainen, 2013; Schröger et al, 2007), unexpected stimulus repetitions (Alain et al, 1999; Horvath & Winkler, 2004; Macdonald & Campbell, 2011; Nordby et al, 1988) and unexpected stimulus omissions (Bendixen et al, 2009; Chennu et al, 2016; SanMiguel et al, 2013; Suda et al, 2022; Yabe et al, 1997).…”

“…Overall, comparable dynamics of brain responses reflecting expectation violation within and across sensory modalities at different scales of complexity are of great interest for a mechanistic understanding of MMRs and fMRI investigations complementing the large body of work done in EEG remain rare. In the current study, we use a tri‐modal version of the roving stimulus paradigm (Grundei et al, 2023) to elicit MMRs in fMRI. The paradigm allows to study responses to rare stimulus transitions independent of their equiprobable features (Baldeweg et al, 2004; Cowan et al, 1993; Garrido et al, 2008).…”