Progressive neural engagement within the IFG-pMTG circuit as gesture and speech entropy and MI advances

Abstract

How information of different modalities (e.g., speech and gesture) frame into a unified one remains unclear in the neurobiology of language. Leveraging the information-theoretic complexity metrics of entropy and mutual information (MI) to quantify gestures and speech, the present event-related potentials and chronometric double-pulse transcranial magnetic stimulation (TMS) study investigated the neural processes during probabilistic gesture-speech integration. In the early processing stage, we found a positive correlation of P1 amplitude with speech entropy and a negative correlation of N1-P2 amplitude with gesture entropy, while the TMS disruption effect of posterior middle temporal gyrus (pMTG) was negatively correlated with speech entropy. In the lexico-semantic processing stage, we observed a negative correlation of N400 amplitude with both gesture entropy and gesture-speech MI, a negative correlation of late positive component amplitude with speech entropy, together with correlations of the TMS interruption effect of inferior frontal gyrus (IFG) with gesture entropy, speech entropy and their MI. Our findings suggest a probabilistically updated lexical representation that was dynamically framed by top-down gesture constrain, bottom-up speech encoding and their interplay. A 5-step process in the pMTG-IFG-pMTG neural circuit is hypothesized for probabilistic gesture-speech processing, which promises a better understanding of the predictive multisensory language processing.

Dynamic neural stages of proportional gesture-speech integration characterized by cross-modal fragment priming

“…Recent research has conceptualized language processing as a sequence of mental states that function quantitively (29) and are stored probabilistically (30). More recently, a study reported that the amplitudes of different ERP components and cortical engagements were proportionally correlated with changes in gesture and speech representations (31). In the present study, by presenting gesture and speech fragments at ve proportionally increased lengths in accordance with their semantic identi cation, increasing nameability was reported.…”

TMS reveals a two-stage priming circuit of gesture-speech integration

“…Both temporal relationships, i.e., gestures leading speech (Kelly et al, 2004;Wu and Coulson, 2005;Holle and Gunter, 2007;Ozyurek et al, 2007;Kelly et al, 2010a;Yap et al, 2011;Pine et al, 2013) and gestures occurring partly ahead of the speech segment (Kelly et al, 2010a;Drijvers et al, 2018;Zhao et al, 2018;Zhao et al, 2021;Zhao et al, 2022), have been examined in previous research. However, the question of how gestures contribute to gesture-speech integration remains unanswered, despite various methodological investigations being carried out (for a review, see Kandana Arachchige et al, 2021).…”

“…Experiment 1 referred to the time course of the integration, where speech onset occurred at 200 ms after the gesture stroke, and the TWs were segmented from the onset of speech (Kelly et al, 2010a;Zhao et al, 2018) (Figure 1A). Experiment 2 referred to the processing stage of gesture-speech information, with speech onset occurred at the DP of gestures while the TWs were divided relative to the IP of speech (Zhao et al, 2021;Zhao et al, 2022) (Figure 1B). Online double-pulse TMS stimulation, a method that is ideal for examining causal brain-behavioral relationships, was applied over either the left IFG or the left pMTG, brain locations believed to be responsible for gesture-speech integration (Zhao et al, 2018;Zhao et al, 2021).…”

TMS reveals a two-stage priming circuit of gesture-speech integration

“…Temporally, gestures have been supposed to occur ahead of Morrel-Samuels and Krauss (1992) or even terminate before ( Fritz et al, 2021 ) their semantic affiliate. Both temporal relationships, i.e., gestures leading speech ( Kelly et al, 2004 ; Wu and Coulson, 2005 ; Holle and Gunter, 2007 ; Ozyurek et al, 2007 ; Kelly et al, 2010a ; Yap et al, 2011 ; Pine et al, 2013 ; ter Bekke et al, 2020 ) and gestures occurring partly ahead of the speech segment ( Kelly et al, 2010a ; Drijvers et al, 2018 ; Zhao et al, 2018 , 2021 , 2022 ), have been examined in previous research. However, the question of how gestures contribute to gesture-speech integration remains unanswered, despite various methodological investigations being carried out ( Kandana Arachchige et al, 2021 ).…”

“…Experiment 1 referred to the time course of the integration, where speech onset occurred at 200 ms after the gesture stroke, and the TWs were segmented from the onset of speech ( Kelly et al, 2010a ; Zhao et al, 2018 ; Figure 1A ). Experiment 2 referred to the processing stage of gesture-speech information, with speech onset occurred at the DP of gestures while the TWs were divided relative to the IP of speech ( Zhao et al, 2021 , 2022 ; Figure 1B ). Online double-pulse TMS stimulation, a method that is ideal for examining causal brain-behavioral relationships, was used in the present study.…”