Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids

Abstract

For people who cannot communicate due to severe paralysis or involuntary movements, technology that decodes intended speech from the brain may offer an alternative means of communication. If decoding proves to be feasible, intracranial Brain-Computer Interface systems can be developed which are designed to translate decoded speech into computer generated speech or to instructions for controlling assistive devices. Recent advances suggest that such decoding may be feasible from sensorimotor cortex, but it is not clear how this challenge can be approached best. One approach is to identify and discriminate elements of spoken language, such as phonemes. We investigated feasibility of decoding four spoken phonemes from the sensorimotor face area, using electrocorticographic signals obtained with high-density electrode grids. Several decoding algorithms including spatiotemporal matched filters, spatial matched filters and support vector machines were compared. Phonemes could be classified correctly at a level of over 75% with spatiotemporal matched filters. Support Vector machine analysis reached a similar level, but spatial matched filters yielded significantly lower scores. The most informative electrodes were clustered along the central sulcus. Highest scores were achieved from time windows centered around voice onset time, but a 500 ms window before onset time could also be classified significantly. The results suggest that phoneme production involves a sequence of robust and reproducible activity patterns on the cortical surface. Importantly, decoding requires inclusion of temporal information to capture the rapid shifts of robust patterns associated with articulator muscle group contraction during production of a phoneme. The high classification scores are likely to be enabled by the use of high density grids, and by the use of discrete phonemes. Implications for use in Brain-Computer Interfaces are discussed.

Differential Representation of Articulatory Gestures and Phonemes in Motor, Premotor, and Inferior Frontal Cortices

“…High gamma activity in ECoG in M1v concurred with Penfield's original somatotopic mappings of the articulators (Penfield and Boldrey, 1937). Several ECoG studies have found evidence that M1v activity roughly correlates with phoneme production (Bouchard et al, 2013;Lotte et al, 2015;Ramsey et al, 2017). Mugler et al demonstrated that single instances of phonemes can be identified during word production using ECoG from M1v and PMv (Mugler et al, 2014b).…”

“…Over the last decade, electrocorticography (ECoG) has enabled identification of neural activity 48 with high spatial and temporal resolution during speech production (Blakely et al, 2008; Bouchard et 49 al., Several ECoG studies have found evidence that M1v activity roughly correlates with phoneme 53 production (Bouchard et al, 2013;Leuthardt et al, 2011;Lotte et al, 2015;Ramsey et al, 2017).…”

Subthalamic nucleus and sensorimotor cortex activity during speech production

“…Functional imaging (fMRI) studies generally provide corroborating evidence for the somatotopic cortical representation of the vocal tract effectors, among other body parts, albeit with a varying degree of overlap among individuals (Lotze et al, 2000;Hesselmann et al, 2004;Pulvermüller et al, 2006;Brown, Ngan, and Liotti, 2007;Meier et al, 2008;Takai, Brown, and Liotti, 2010;Carey et al, 2017). Recently, ECoG studies have elaborated the notion of cortical articulatory somatotopy by revealing differentiated neural representations for fine-grained phonetic features and complex kinematics underlying speech articulation (Bouchard et al, 2013;Bouchard & Chang, 2014;Mugler et al, 2014;Lotte et al, 2015;Bouchard et al, 2016;Cheung et al, 2016;Ramsey et al, 2017;Chartier et al, 2018;Conant et al, 2018).…”

Speech Synthesis from ECoG using Densely Connected 3D Convolutional Neural Networks

“…The spatio-temporal dynamics of word retrieval during speech production was shown in [6]. Other approaches demonstrated that speech can be decoded from invasively measured brain activity such as words [7], phonemes [8,9], phonetic features [10,11], articulatory gestures [12] and continuous sentences [13,14].…”