Combining predictive coding with neural oscillations optimizes on-line speech processing

Abstract

Speech comprehension requires segmenting continuous speech to connect it on-line with discrete linguistic neural representations. This process relies on theta-gamma oscillation coupling, which tracks syllables and encodes them in decipherable neural activity. Speech comprehension also strongly depends on contextual cues predicting speech structure and content. To explore the effects of theta-gamma coupling on bottom-up/top-down dynamics during on-line speech perception, we designed a generative model that can recognize syllable sequences in continuous speech. The model uses theta oscillations to detect syllable onsets and align both gamma-rate encoding activity with syllable boundaries and predictions with speech input. We observed that the model performed best when theta oscillations were used to align gamma units with input syllables, i.e. when bidirectional information flows were coordinated, and internal timing knowledge was exploited. This work demonstrates that notions of predictive coding and neural oscillations can usefully be brought together to account for dynamic on-line sensory processing.

Processing rhythm in speech and music: Shared mechanisms and implications for developmental speech and language disorders.

“…Observing neural oscillations at hierarchical levels not physically present in the stimulus is particularly pertinent to the discussion of whether neural oscillations represent the entrainment of already-present endogenous oscillations to an external stimulus ( entrainment in the narrow sense , Obleser & Kayser, 2019), or whether they only represent evoked neural responses to the acoustic (rhythmic) properties of the external stimulus (see Haegens, 2020; Haegens & Zion Golumbic, 2018; and Zoefel et al, 2018, for discussion). This distinction has implications for the active role of neural oscillations in the prediction of upcoming events via the entrainment of self-sustaining endogenous oscillations (see also links with predictive coding, Friston, 2018; Giraud & Arnal, 2018; Hovsepyan et al, 2018; Rao & Ballard, 1999). Despite an ongoing debate, accumulating evidence suggests that observed neural oscillations cannot be reduced to evoked responses, but include also the entrainment of neural oscillations with functional significance (e.g., Doelling et al, 2019; van Bree et al, 2021).…”

Parcels and particles: Markov blankets in the brain

“…Usually, these asymmetries are expressed in terms of things like laminar specificity that distinguish between forward and backward connections (Buffalo, Fries, Landman, Buschman, & Desimone, 2011 ; Grossberg, 2007 ; Haeusler & Maass, 2007 ; Hilgetag, O’Neill, & Young, 2000 ; Thomson & Bannister, 2003 ; Trojanowski & Jacobson, 1977 ). More recently, asymmetries in spectral content have become an emerging theme (Arnal & Giraud, 2012 ; Bastos et al, 2015 ; Buffalo et al, 2011 ; Giraud & Poeppel, 2012 ; Hovsepyan, Olasagasti, & Giraud, 2018 ; Self, van Kerkoerle, Goebel, & Roelfsema, 2019 ; Singer, Sejnowski, & Rakic, 2019 ; van Kerkoerle et al, 2014 ).…”

Syllable Inference as a Mechanism for Spoken Language Understanding

“…) Our proposal that a syllable-level of modeling underlies neural processes for determining lexical structures in speech draws on linguistic notions that syllables are universal units of language phonology composed of an obligatory "vocalic" or vowel nucleus (Goldsmith, 2011). The SI account ties the arrival of vowel-related amplitude rise cues (Oganian & Chang, 2019) to internal predictive processes of syllable-and word-modeling (Donhauser & Baillet, 2020;Martin, 2020), providing a novel account of how neural oscillations may dually reflect integration of exogenous signal cues and endogenous generative representation-building (e.g., language hierarchical structures) (Hovsepyan, Olasagasti, & Giraud, 2020;Klimovich-Gray & Molinaro, 2020). The inherent asymmetry of vowels and consonants--vowels have statistically more energy and more open vocal cavity positions than consonants (Stevens, 1999)--further suggests a linking hypothesis such that domain-general processes could plausibly be deployed for language learning of rate-varying auditory information (Bosker, 2017;Háden et al, 2015;Martin, 2016;Nazzi & Cutler, 2019).…”