Objective. In order to increase the number of states classified by a brain-computer interface (BCI), we utilized a motor imagery task where subjects imagined both force and speed of hand clenching. Approach. The BCI utilized simultaneously recorded electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) signals. The time-phase-frequency feature was extracted from EEG, whereas the HbD [the difference of oxy-hemoglobin (HbO) and deoxyhemoglobin (Hb)] feature was used to improve the classification accuracy of fNIRS. The EEG and fNIRS features were combined and optimized using the joint mutual information (JMI) feature selection criterion; then the extracted features were classified with the extreme learning machines (ELMs). Main results. In this study, the averaged classification accuracy of EEG signals achieved by the time-phase-frequency feature improved by 7%, to 18%, more than the single-type feature, and improved by 15% more than common spatial pattern (CSP) feature. The HbD feature of fNIRS signals improved the accuracy by 1%, to 4%, more than Hb, HbO, or HbT (total hemoglobin). The EEG-fNIRS feature for decoding motor imagery of both force and speed of hand clenching achieved an accuracy of 89% ± 2%, and improved the accuracy by 1% to 5% more than the sole EEG or fNIRS feature. Significance. Our novel motor imagery paradigm improves BCI performance by increasing the number of extracted commands. Both the time-phase-frequency and the HbD feature improve the classification accuracy of EEG and fNIRS signals, respectively, and the hybrid EEG-fNIRS technique achieves a higher decoding accuracy for two-class motor imagery, which may provide the framework for future multi-modal online BCI systems.

Functional near-infrared spectroscopy (fNIRS) is an emerging optical technique, which can assess brain activities associated with tasks. In this study, six participants were asked to perform three imageries of hand clenching associated with force and speed, respectively. Joint mutual information (JMI) criterion was used to extract the optimal features of hemodynamic responses. And extreme learning machine (ELM) was employed to be the classifier. ELM solved the major bottleneck of feedforward neural networks in learning speed, this classifier was easily implemented and less sensitive to specified parameters. The 2-class fNIRS-BCI system was firstly built with an average accuracy of 76.7 %, when all force and speed tasks were categorized as one class, respectively. The multi-class systems based on different levels of force and speed attempted to be investigated, the accuracies were moderate. This study provided a novel paradigm for establishing fNIRS-BCI system, and provided a possibility to produce more degrees of freedom in BCI system.