Adaptive multi-degree of freedom Brain Computer Interface using online feedback: Towards novel methods and metrics of mutual adaptation between humans and machines for BCI

Abstract

This paper proposes a novel adaptive online-feedback methodology for Brain Computer Interfaces (BCI). The method uses ElectroEncephaloGraphic (EEG) signals and combines motor with speech imagery to allow for tasks that involve multiple degrees of freedom (DoF). The main approach utilizes the covariance matrix descriptor as feature, and the Relevance Vector Machines (RVM) classifier. The novel contributions include, (1) a new method to select representative data to update the RVM model, and (2) an online classifier which is an adaptively-weighted mixture of RVM models to account for the users’ exploration and exploitation processes during the learning phase. Instead of evaluating the subjects’ performance solely based on the conventional metric of accuracy, we analyze their skill’s improvement based on 3 other criteria, namely the confusion matrix’s quality, the separability of the data, and their instability. After collecting calibration data for 8 minutes in the first run, 8 participants were able to control the system while receiving visual feedback in the subsequent runs. We observed significant improvement in all subjects, including two of them who fell into the BCI illiteracy category. Our proposed BCI system complements the existing approaches in several aspects. First, the co-adaptation paradigm not only adapts the classifiers, but also allows the users to actively discover their own way to use the BCI through their exploration and exploitation processes. Furthermore, the auto-calibrating system can be used immediately with a minimal calibration time. Finally, this is the first work to combine motor and speech imagery in an online feedback experiment to provide multiple DoF for BCI control applications.

Co-Adaptive Control of Bionic Limbs via Unsupervised Adaptation of Muscle Synergies

“…Furthermore, under the broader umbrella of human-machine interfacing, the brain-computer interface (BCI) domain has similarly explored co-adaptive methods to improve controller performance [41], [42]. For example, in classification-based BCI, changes in control signal separability have been used as evidence for proficiency gain [43]. Nonetheless, in this work, the simultaneous adaptation of user and machine is implicit, given that feedback from model changes are instantaneous and have a direct influence on the errordriven dynamics of the user's motor adaptation.…”

Vocal Imagery vs Intention: Viability of Vocal-Based EEG-BCI Paradigms

“…In another study, Brumberg et.al., proposed a MI BCI system with real-time auditory and visual feedback of speech sounds (for production of three vowels, /i/, /a/, and /u/) that has the potential to serve as a communication tool for individuals with severe neuromotor impairments [21]. Most recently, Nguyen et.al., [22] combined MI and speech imagery for an online BCI control, to allow for tasks that involve multiple degrees of freedom. Dash et al, investigated speaker-independent neural speech decoding of five continuous phrases from MEG signals while the subjects produced speech covertly (imagination) or overtly (articulation) and showed significantly improved decoding performance in their method [23].…”

Neural decoding schemes for BCIs in Neuroergonomics

“…The current state-of-the-art techniques, in the field of BCIs, include among others Riemannian geometry-based classifiers [Kal+19], filter banks [KST17], adaptive classifiers [NKA19] and graph signal processing [RNC16]. On top of them, and in complete harmony with the current trends in empirical data analysis, deep learning has come into the picture.…”

“…The applicability of surface electromyography (EMG) in real-time driving fatigue detection is limited [SA18]. EEG has been considered as a promising modality for driving fatigue detection, owing to its high temporal resolution, high portability, and good sensitivity to brain state [NKA19]. In particular, EEG can be used to non-invasively measure the neuronal electrical activity from the scalp surface to provide a direct assessment of brain fatigue status [Che+17].…”

“…Researchers within the computational neuroscience and machine learning communities have put a lot of effort into developing signal processing techniques and computational intelligence algorithms that perform robust brain decoding from EEG signals, despite the aforementioned challenges. The current state-of-the-art techniques include among others Riemannian geometry-based classifiers [Kal+19], filter banks [KST17], adaptive classifiers [NKA19] and graph signal processing [RNC16]. On top of them, and in complete harmony with the current trends in empirical data analysis, deep learning has come into the picture.…”