Decoding lexical tones and vowels in imagined tonal monosyllables using fNIRS signals

Abstract

Objective. Speech is a common way of communication. Decoding verbal intent could provide a naturalistic communication way for people with severe motor disabilities. Active brain computer interaction (BCI) speller is one of the most commonly used speech BCIs. To reduce the spelling time of Chinese words, identifying vowels and tones that are embedded in imagined Chinese words is essential. Functional near-infrared spectroscopy (fNIRS) has been widely used in BCI because it is portable, non-invasive, safe, low cost, and has a relatively high spatial resolution. Approach. In this study, an active BCI speller based on fNIRS is presented by covertly rehearsing tonal monosyllables with vowels (i.e., /a/, /i/, /o/, and /u/) and four lexical tones in Mandarin Chinese (i.e., tones 1, 2, 3, and 4) for 10 s. Main results. fNIRS results showed significant differences in the right superior temporal gyrus between imagined vowels with tone 2/3/4 and those with tone 1 (i.e., more activations and stronger connections to other brain regions for imagined vowels with tones 2/3/4 than for those with tone 1). Speech-related areas for tone imagery (i.e., the right hemisphere) provided majority of information for identifying tones, while the left hemisphere had advantages in vowel identification. Having decoded both vowels and tones during the post-stimulus 15 s period, the average classification accuracies exceeded 40 % and 70 % in multiclass (i.e., four classes) and binary settings, respectively. To spell words more quickly, the time window size for decoding was reduced from 15 s to 2.5 s while the classification accuracies were not significantly reduced. Significance. For the first time, this work demonstrated the possibility of discriminating lexical tones and vowels in imagined tonal syllables. In addition, the reduced time window for decoding indicated that the spelling time of Chinese words could be significantly reduced in the fNIRS-based BCIs.

Neural signatures of imaginary motivational states: desire for music, movement and social play

“…However, the asymmetry was more pronounced for music imagery with a greater magnitude of N400 sources in the right hemisphere than the left. Consistently, previous studies have reported a right hemispheric asymmetry for tone imagery (Guo and Chen 2022), visuomotor imagery (Kwon et al 2023), spatial navigation imagery (Boly et al 2007), facial expression imagery (Kim et al 2007), emotional imagery (Tomasino et al 2014) and music imagery (Zatorre and Samson 1991;Zatorre and Halpern 1993;Halpern 2001). On the flip side, a tendency towards left hemispheric asymmetry would be usual for imagining movement (Zou et al 2022), imagining written language (see the review by Liu et al 2022) or tools (Belardinelli et al 2009).…”

Impacts of simplifying articulation movements imagery to speech imagery BCI performance

“…Recently many SI BCIs based on fNIRS have been developed to build user-friendly and naturalistic communication systems [12][13][14][15][16]. Several studies detected the answers to yes or no questions of individuals who covertly repeated 'yes' or 'no' with fNIRS and the binary classification accuracies of those studies were reported to range from 70% to 76% [e.g.…”

“…In order to be more in line with the practical application of BCI systems, Sereshkeh et al introduced a ternary BCI based on fNIRS to distinguish covertly rehearsed yes or no responses and an equivalent duration of unconstrained rest in a sample of 12 participants, and reported an online 3-class classification accuracy of 64% [15]. Four vowels and four lexical tones that are embedded in imagined tonal syllables were discriminated based on fNIRS, the four-class classification accuracies of imagined vowels and tones were 40% and 41%, respectively [16]. Although SI BCIs based on fNIRS have been developed for many years, their decoding performances still need further improvement.…”