Hearing and seeing meaning in noise: Alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech comprehension

Abstract

During face‐to‐face communication, listeners integrate speech with gestures. The semantic information conveyed by iconic gestures (e.g., a drinking gesture) can aid speech comprehension in adverse listening conditions. In this magnetoencephalography (MEG) study, we investigated the spatiotemporal neural oscillatory activity associated with gestural enhancement of degraded speech comprehension. Participants watched videos of an actress uttering clear or degraded speech, accompanied by a gesture or not and completed a cued‐recall task after watching every video. When gestures semantically disambiguated degraded speech comprehension, an alpha and beta power suppression and a gamma power increase revealed engagement and active processing in the hand‐area of the motor cortex, the extended language network (LIFG/pSTS/STG/MTG), medial temporal lobe, and occipital regions. These observed low‐ and high‐frequency oscillatory modulations in these areas support general unification, integration and lexical access processes during online language comprehension, and simulation of and increased visual attention to manual gestures over time. All individual oscillatory power modulations associated with gestural enhancement of degraded speech comprehension predicted a listener's correct disambiguation of the degraded verb after watching the videos. Our results thus go beyond the previously proposed role of oscillatory dynamics in unimodal degraded speech comprehension and provide first evidence for the role of low‐ and high‐frequency oscillations in predicting the integration of auditory and visual information at a semantic level.

Degree of Language Experience Modulates Visual Attention to Visible Speech and Iconic Gestures During Clear and Degraded Speech Comprehension

“…In line with our hypotheses, both native and non‐native listeners demonstrated a clear gestural enhancement effect on the cued–recall task (following Drijvers & Özyürek, , ; Drijvers et al, , ). This gestural enhancement effect was the largest when speech was degraded in both native and non‐native listeners.…”

“…This N400 effect in clear speech was larger for non‐native listeners than native listeners, which might indicate that they focus more strongly on gestures than native listeners, to extract semantic information to aid comprehension. Similarly, previous neuroimaging work has indicated that both native and non‐native listeners engage their visual cortex more when speech is degraded and a gesture is present than when speech is clear or no gesture is present, possibly to allocate more visual attention to gestures and increase information uptake (Drijvers et al, , ). Non‐native listeners, however, engage areas involved in semantic retrieval and semantic unification, and visible speech processing less than native listeners during gestural enhancement of degraded speech, suggesting that non‐native listeners might be hindered in integrating the degraded phonological cues with the semantic information conveyed by the gesture.…”

“…The stimuli that were used in this experiment were the same stimuli as described in Drijvers et al (). Participants were presented with 160 video clips in which a non‐professional actress uttered a highly frequent Dutch action verbs in clear or degraded speech, while she performed an iconic gesture or not.…”

“…Visible speech, which consists of movements of tongue, teeth and lip movements, has been shown to enhance clear and degraded speech comprehension for both native and non‐native listeners (Erber, ; Munhall, ; Navarra & Soto‐Faraco, ; Ross, Saint‐Amour, Leavitt, Javitt, & Foxe, ; Sumby & Pollock, ), as it can provide phonological information about the speech signal that can enhance comprehension. Next to visible speech, iconic hand gestures, which can convey semantic information about object attributes, actions, and space (McNeill, ), can also enhance clear and degraded speech comprehension (Drijvers, Özyürek, & Jensen, ; Holle, Obleser, Rueschemeyer, & Gunter, ), especially in a joint context with visible speech (Drijvers & Özyürek, ), and for both native and non‐native speakers (Drijvers & Özyürek, ; Drijvers, van der Plas, Özyürek, & Jensen, ). However, little is known about how native and non‐native listeners allocate their visual attention to benefit from these inputs during speech comprehension in a joint context, especially in adverse listening conditions, such as when speech is degraded.…”

Direct neurophysiological evidence for a role of the human anterior cingulate cortex in central command

“…A 4A Hann window of 2s in width was selected so that the signal could be carefully examined. The area under each power spectrum for each condition was calculated for the following frequency bands: 4-8Hz (θ), 8-12Hz (α), 12-25Hz (β), 25-60Hz (low γ) and 60-90Hz (high γ) (26, 27). …”