Mapping Acoustics to Articulatory Gestures in Dutch: Relating Speech Gestures, Acoustics and Neural Data

Abstract

Completely locked-in patients suffer from paralysis affecting every muscle in their body, reducing their communication means to brain-computer interfaces (BCIs). State-of-the-art BCIs have a slow spelling rate, which inevitably places a burden on patients' quality of life. Novel techniques address this problem by following a bio-mimetic approach, which consists of decoding sensory-motor cortex (SMC) activity that underlies the movements of the vocal tract's articulators. As recording articulatory data in combination with neural recordings is often unfeasible, the goal of this study was to develop an acoustic-to-articulatory inversion (AAI) model, i.e. an algorithm that generates articulatory data (speech gestures) from acoustics. A fully convolutional neural network was trained to solve the AAI mapping, and was tested on an unseen acoustic set, recorded simultaneously with neural data. Representational similarity analysis was then used to assess the relationship between predicted gestures and neural responses. The network's predictions and targets were significantly correlated. Moreover, SMC neural activity was correlated to the vocal tract gestural dynamics. The present AAI model has the potential to further our understanding of the relationship between neural, gestural and acoustic signals and lay the foundations for the development of a bio-mimetic speech BCI.Clinical relevance-This study investigates the relationship between articulatory gestures during speech and the underlying neural activity. The topic is central for development of braincomputer interfaces for severely paralysed individuals.

Comparison of acoustic-to-articulatory and brain-to-articulatory mapping during speech production using ultrasound tongue imaging and EEG

“…In Sec. 1, we have shown several previous approaches for brainto-speech synthesis with articulatory information included; but such articulation was always indirectly measured / estimated using acoustic-to-articulatory inversion, and not recorded in parallel with brain-related data [22,27,28,29,30]. As suggested by the papers above, an obvious solution for speech BCIs is the examination of articulation as an intermediate representation between the brain signal and the resulting final speech, which we dealt with in [26] and extended in this article with ultrasound-based acoustic-to-articulatory inversion.…”

“…Besides, the other related studies all use estimated articulatory data, i.e. they take into account the articulatory information inferred from the speech signal or from textual contents (e.g., [27,22,28]. Several studies appeared this year which have the aim to predict articulatory-related information from the brain signal.…”

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models

“…However, for development of such a speech decoding approach, articulator movements need to be recorded during speech simultaneously with the brain activity, which is rarely feasible in ECoG research due to the additional burden it would inflict on the study participants. Acoustic-toarticulatory inversion models can potentially be used to facilitate decoding and sound reconstruction using kinematic traces [18,50]. Otherwise, another system will be required to map the decoded kinematic movements to language information, such as phonemes, syllables or words [51].…”