Self-Supervised Learning of Neural Speech Representations From Unlabeled Intracranial Signals

Abstract

Neuroprosthetics have demonstrated the potential to decode speech from intracranial brain signals, and hold promise for one day returning the ability to speak to those who have lost it. However, data in this domain is scarce, highly variable, and costly to label for supervised modeling. In order to address these constraints, we present brain2vec, a transformer-based approach for learning feature representations from intracranial electroencephalogram data. Brain2vec combines a self-supervised learning methodology, neuroanatomical positional embeddings, and the contextual representations of transformers to achieve three novelties: (1) learning from unlabeled intracranial brain signals, (2) learning from multiple participants simultaneously, all while (3) utilizing only raw unprocessed data. To assess our approach, we use a leave-one-participant-out validation procedure to separate brain2vec's feature learning from the holdout participant's speech-related supervised classification tasks. With only two linear layers, we achieve 90% accuracy on a canonical speech detection task, 42% accuracy on a more challenging 4-class speech-related behavior recognition, and 53% accuracy when applied to a 10-class, few-shot word classification task. Combined with visualizations of unsupervised class separation in the learned features, our results evidence brain2vec's ability to learn highly generalized representations of neural activity without the need for labels or consistent sensor location.

Speech synthesis from intracranial stereotactic Electroencephalography using a neural vocoder

“…Both the subjective listening tests and objective evaluations show that the neural network-based approaches outperformed the linear regression baseline. The relatively low naturalness scores (18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29) indicate that sEEG-based synthesized speech is far from being intelligible, but clearly, has properties similar to the natural speech signal, both visually on the spectrograms, and when listening to the samples. Therefore, we expect that our results might help future speech-based Brain-Computer Interfaces.…”

“…Another recent article, Lesaja et al [24] presents brain2vec, a self-supervised model for learning speech-related hidden unit representations from unlabeled intracranial EEG data. Brain2vec's performance rivals that of competitive supervised learning methods on speech activity detection and word classification tasks, indicating potential practical applications in speech decoding using intracranial EEG data.…”

“…As a summary of the evaluation, the objective MCD score was not always found to be helpful in our case (i.e, it does not highly correspond to the correlations of [7]), but clearly, the subjective listening test could show the differences between the speakers of low and high correlation. The relatively low naturalness scores (18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29) indicate that sEEG-based synthesized speech is far from being intelligible, but at least, has properties similar to the natural speech signal.…”