Online classification of imagined speech using functional near-infrared spectroscopy signals

Abstract

Objective. Most brain–computer interfaces (BCIs) based on functional near-infrared spectroscopy (fNIRS) require that users perform mental tasks such as motor imagery, mental arithmetic, or music imagery to convey a message or to answer simple yes or no questions. These cognitive tasks usually have no direct association with the communicative intent, which makes them difficult for users to perform. Approach. In this paper, a 3-class intuitive BCI is presented which enables users to directly answer yes or no questions by covertly rehearsing the word ‘yes’ or ‘no’ for 15 s. The BCI also admits an equivalent duration of unconstrained rest which constitutes the third discernable task. Twelve participants each completed one offline block and six online blocks over the course of two sessions. The mean value of the change in oxygenated hemoglobin concentration during a trial was calculated for each channel and used to train a regularized linear discriminant analysis (RLDA) classifier. Main results. By the final online block, nine out of 12 participants were performing above chance (p < 0.001 using the binomial cumulative distribution), with a 3-class accuracy of 83.8% ± 9.4%. Even when considering all participants, the average online 3-class accuracy over the last three blocks was 64.1 % ± 20.6%, with only three participants scoring below chance (p < 0.001). For most participants, channels in the left temporal and temporoparietal cortex provided the most discriminative information. Significance. To our knowledge, this is the first report of an online 3-class imagined speech BCI. Our findings suggest that imagined speech can be used as a reliable activation task for selected users for development of more intuitive BCIs for communication.

Brain-Based Binary Communication Using Spatiotemporal Features of fNIRS Responses

“…In line with this, it has been demonstrated that HbO signal is more robust than HbR for motor imagery specific activation (Mihara et al, 2012). Likewise, Rezazadeh Sereshkeh et al (2018) reported that HbO signals yielded the highest accuracies in their 3-class BCI using imagined speech, and Hwang et al (2016) reported that HbO features yield more discriminative information than HbR features in 2-class communication.…”

“…However, to our knowledge, no study has tested the use of two mental tasks directly in a communication experiment. In a recent study, participants imagined different mental speech content for answering yes/no questions intuitively, i.e., imagining saying "yes" or "no" repeatedly (Rezazadeh Sereshkeh et al, 2018). An average accuracy of 64.1% was attained over two experimental sessions.…”

“…In the context of communication, most hemodynamic BCI systems rely on mental imagery for intentional generation of brain signals. Commonly used mental imagery tasks (see Naseer and Hong, 2015, for a more extensive review) include mental speech (Sorger et al, 2012;Rezazadeh Sereshkeh et al, 2018), mental calculation/counting (Naito et al, 2007;Power et al, 2012;Sorger et al, 2009Sorger et al, , 2012 and motor imagery (Coyle et al, 2007;Sorger et al, 2009Sorger et al, , 2012Monti et al, 2010;Abdalmalak et al, 2017a). Most fNIRS based-BCI communication studies focus on binary communication, as multi-class fNIRS-based BCIs are not yet enabling effective BCI control (Power et al, 2012;Weyand and Chau, 2015).…”

“…Additionally, a multivariate approach was applied to explore the discernibility of the two tasks based on their spatial brain activation patterns only. Comfort ratings were obtained throughout the experiment as other studies have reported that participants may withdraw from fNIRS recordings due to headset discomfort (Suzuki et al, 2010;Cui et al, 2011;Rezazadeh Sereshkeh et al, 2018). In addition, we evaluated whether the presence of specific physical features of participants (e.g., hair thickness, root density, or color) affected fNIRSsignal quality and subsequent decoding results (Koizumi et al, 1999;Coyle et al, 2005;Cui et al, 2011;Khan et al, 2012;Fang et al, 2018).…”

Performance Improvement of Near-Infrared Spectroscopy-Based Brain-Computer Interface Using Regularized Linear Discriminant Analysis Ensemble Classifier Based on Bootstrap Aggregating

“…Among them, linear discriminant analysis (LDA) classifier has been most widely used for NIRS-BCIs because of its excellent performance reflected by both a fast learning rate and a good classification performance (Holper and Wolf, 2011;Power et al, 2011Power et al, , 2012aSchudlo and Chau, 2014;Hong et al, 2015;Shin et al, 2016Shin et al, , 2018aHong and Khan, 2017). In applying the classifier, dimension reduction or feature selection methods are generally employed because the number of NIRS feature vectors is usually larger than that of training datasets and this might degrade the BCI performance due to the poor empirical sample covariance Sereshkeh et al, 2019). Regularization with a shrinkage parameter can be another option to alleviate the adverse effect of the large dimensionality (Fazli et al, 2012).…”