Decoding and synthesizing tonal language speech from brain activity

Liu, Yan; Zhao, Zehao; Xu, Minpeng; Yu, Haiqing; Zhu, Yanming; Zhang, Jie; Bu, Linghao; Zhang, Xiaoluo; Lu, Junfeng; Li, Yuanning; Ming, Dong; Wu, Jinsong

doi:10.1126/sciadv.adh0478

Cited by 14 publications

(14 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tone, a crucial aspect of Mandarin syllables, differentiates both lexical and grammatical meanings through variations in pitch carried by the finals 21,28 . We visualized the pitch changes over time for the four Mandarin tones (Fig.…”

Section: Resultsmentioning

confidence: 99%

Acoustic inspired brain-to-sentence decoder for logosyllabic language

Feng,

Cao,

et al. 2023

Preprint

View full text Add to dashboard Cite

Many severe neurological diseases, such as stroke and amyotrophic lateral sclerosis, can lead to patients completely losing their ability to communicate1,2. Several language brain-computer interface systems have been shown to have the potential to help these patients regain their communication abilities by decoding speech or movement-related neural signals3. However, the aforementioned language brain-computer interfaces (BCIs) were all developed based on alphabetic linguistic systems without one specially designed for logosyllabic languages like Mandarin Chinese. Here, we established the first language BCI specifically designed for Chinese, decoding speech-related stereoelectroencephalography (sEEG) signals into sentences. Firstly, based on the acoustic features of full-spectrum Chinese syllable pronunciation, we constructed prediction models for three syllable elements (initials, tones, and finals). Subsequently, a language model transformed all the predicted syllable elements, integrating them with semantic information, to generate the most probable sentence. Ultimately, we achieved a high-performance decoder with a median character error rate of 29%, demonstrating preliminary potential for clinical application. Our research fills the gap in language BCI for logosyllabic languages and leverages a powerful language model to enhance the performance of the language BCI, offering new insights for future logosyllabic language neuroprosthesis studies.

show abstract

Section: Resultsmentioning

confidence: 99%

Acoustic inspired brain-to-sentence decoder for logosyllabic language

Feng,

Cao,

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, all the patient articulated the tone of "不" in "不计得失" with a pitch trajectory more similar to tone 2 rather than tone 4 in single syllable form or in other phrases such as "不得不", due to the rule of tone sandhi (Fig 4a Similarly, given only monosyllabic information, the tone decoder would perform suboptimal. To show this, we adopted a baseline monosyllable decoder model previously used in Liu et al 15 . The monosyllable decoder only took in the neural activity aligned to the current syllable utterance and did not consider contextual syllables.…”

Section: Robustness Of the Speech Decoder Under Tonal Variance In Nat...mentioning

confidence: 99%

“…The distinct feature of these languages is the use of pitch to distinguish lexical and grammatical meaning. While prior research has investigated decoding stereotypical instances of lexical tones from neural activity for monosyllabic speech 15 , decoding continuous tonal sentences is still a challenging issue. Unlike the relatively stable acoustic cues in canonical forms, natural speech introduces substantial variability in tone components.…”

Section: Introductionmentioning

confidence: 99%

A brain-to-text framework of decoding natural tonal sentences

Zhang,

Wang,

Qian

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Speech brain-computer interfaces (BCIs) directly translate brain activity into speech sound and text, yet decoding tonal languages like Mandarin Chinese poses a significant, unexplored challenge. Despite successful cases in non-tonal languages, the complexities of Mandarin, with its distinct syllabic structures and pivotal lexical information conveyed through tonal nuances, present challenges in BCI decoding. Here we designed a brain-to-text framework to decode Mandarin tonal sentences from invasive neural recordings. Our modular approach dissects speech onset, base syllables, and lexical tones, integrating them with contextual information through Bayesian likelihood and the Viterbi decoder. The results demonstrate accurate tone and syllable decoding under variances in continuous naturalistic speech production, surpassing previous intracranial Mandarin tonal syllable decoders in decoding accuracy. We also verified the robustness of our decoding framework and showed that the model hyperparameters can be generalized across participants of varied gender, age, education backgrounds, pronunciation behaviors, and coverage of electrodes. Our pilot study shed lights on the feasibility of more generalizable brain-to-text decoding of natural tonal sentences from patients with high heterogeneities.

show abstract

“…Therefore, any adjustments to acoustic predictions must be applied in real time, for example, through parallel phonetic or prosodic predictions. In the latter case, it would be conceivable to apply real-time pitch shifting to create artificial tones [64] or intonation patterns [46] that carry additional meaning or emotion. Alternatively, the continuous estimation of intermediate articulatory features, fed into a physical model of the vocal tract, would likely generate more natural-sounding speech outputs [50].…”

Section: Minimal Feedback Delay Will Improve Sense Of Agencymentioning

confidence: 99%

Continuous synthesis of artificial speech sounds from human cortical surface recordings during silent speech production

et al. 2023

View full text Add to dashboard Cite

Objective: Brain-computer interfaces can restore various forms of communication in paralyzed patients who have lost their ability to articulate intelligible speech. This study aimed to demonstrate the feasibility of closed-loop synthesis of artificial speech sounds from human cortical surface recordings during silent speech production. Approach: Ten participants with intractable epilepsy were temporarily implanted with intracranial electrode arrays over cortical surfaces. A decoding model that predicted audible outputs directly from patient-specific neural feature inputs was trained during overt word reading and immediately tested with overt, mimed and imagined word reading. Predicted outputs were later assessed objectively against corresponding voice recordings and subjectively through human perceptual judgments. Main results: Artificial speech sounds were successfully synthesized during overt and mimed utterances by two participants with some coverage of the precentral gyrus. About a third of these sounds were correctly identified by naïve listeners in two-alternative forced-choice tasks. A similar outcome could not be achieved during imagined utterances by any of the participants. However, neural feature contribution analyses suggested the presence of exploitable activation patterns during imagined speech in the postcentral gyrus and the superior temporal gyrus. In future work, a more comprehensive coverage of cortical surfaces, including posterior parts of the middle frontal gyrus and the inferior frontal gyrus, could improve synthesis performance during imagined speech. Significance: As the field of speech neuroprostheses is rapidly moving toward clinical trials, this study addressed important considerations about task instructions and brain coverage when conducting research on silent speech with non-target participants.

show abstract

Decoding and synthesizing tonal language speech from brain activity

Cited by 14 publications

References 35 publications

Acoustic inspired brain-to-sentence decoder for logosyllabic language

Acoustic inspired brain-to-sentence decoder for logosyllabic language

A brain-to-text framework of decoding natural tonal sentences

Continuous synthesis of artificial speech sounds from human cortical surface recordings during silent speech production

Contact Info

Product

Resources

About