Gait adaptation to visual kinematic perturbations using a real-time closed-loop brain–computer interface to a virtual reality avatar

Luu, Trieu Phat; He, Yongtian; Brown, Samuel M.; Nakagame, Sho; Contreras-Vidal, José L.

doi:10.1088/1741-2560/13/3/036006

Cited by 95 publications

(69 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pearson's correlation coefficient (r-value) was used in our previous studies to measure performance 18,25 .…”

Section: Metricsmentioning

confidence: 99%

See 1 more Smart Citation

An empirical comparison of neural networks and machine learning algorithms for EEG gait decoding

Nakagome

Luu

et al. 2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

previous studies of Brain computer interfaces (Bci) based on scalp electroencephalography (eeG) have demonstrated the feasibility of decoding kinematics for lower limb movements during walking. in this computational study, we investigated offline decoding analysis with different models and conditions to assess how they influence the performance and stability of the decoder. Specifically, we conducted three computational decoding experiments that investigated decoding accuracy: (1) based on delta band time-domain features, (2) when downsampling data, (3) of different frequency band features. In each experiment, eight different decoder algorithms were compared including the current stateof-the-art. Different tap sizes (sample window sizes) were also evaluated for a real-time applicability assessment. A feature of importance analysis was conducted to ascertain which features were most relevant for decoding; moreover, the stability to perturbations was assessed to quantify the robustness of the methods. Results indicated that generally the Gated Recurrent Unit (GRU) and Quasi Recurrent neural network (QRnn) outperformed other methods in terms of decoding accuracy and stability. previous state-of-the-art Unscented Kalman filter (UKf) still outperformed other decoders when using smaller tap sizes, with fast convergence in performance, but occurred at a cost to noise vulnerability. Downsampling and the inclusion of other frequency band features yielded overall improvement in performance. the results suggest that neural network-based decoders with downsampling or a wide range of frequency band features could not only improve decoder performance but also robustness with applications for stable use of Bcis.Brain Computer Interfaces (BCI) record, infer and translate different parameters associated with movement from different types of brain signals to provide volitional control to prosthetic limbs, exoskeletons, computers, and even digital avatars. The part of the BCI which deciphers the user's motor intent from recorded brain activity is typically referred to as a neural decoder. Building high-performance neural decoders is important in four different aspects: (1) usability, (2) salient feature identification and quantification, (3) understanding of the underlying neural representations 1 , and as (4) a potential metric of neural function. First, BCI neural decoders based on scalp electroencephalography (EEG) are being designed for assistive and therapeutical applications for patients with motor disabilities in order to promote plasticity and facilitate rehabilitation 2,3 . Thus, higher accuracy in decoding performance determines the usability of the system 4 . Second, many neural features (e.g., time and frequency domain features, channel locations, channel and source domain features, to name a few 5,6 ) are likely to contain varying information about motor intent and thus are candidates for decoding human movement. However, it is often difficult to identify and quantify important features given the complexities of perfor...

show abstract

“…Pearson's correlation coefficient (r-value) was used in our previous studies to measure performance 18,25 .…”

Section: Metricsmentioning

confidence: 99%

“…Experiment 1: Decoding based on delta band features. The protocol for Experiment 1 is equivalent to the real-time decoding pipeline used in the previous studies 18,25 . This is the baseline data processing pipeline, which will be used as a comparison for the following two experiments.…”

Section: Metricsmentioning

confidence: 99%

An empirical comparison of neural networks and machine learning algorithms for EEG gait decoding

Nakagome

Luu

et al. 2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

show abstract

“…Previous studies have shown that low delta band (0.1–2 Hz) EEG contains intended movement-related information for decoding the kinematics of lower limb or gait states (Presacco et al, 2011, 2012; Jorquera et al, 2013; Kilicarslan et al, 2013; Bulea et al, 2014; Luu et al, 2016). For example, in Presacco et al (2011), Presacco et al (2012), and Luu et al (2016), it was shown that delta band EEG contains information about gait movement kinematics that can be decoded using Wiener or Kalman filters. In Kilicarslan et al (2013), Jorquera et al (2013), and Bulea et al (2014), it was shown that movement-type (e.g., “stop,” “go,” etc.)…”

Section: Introductionmentioning

confidence: 99%

Multiple Kernel Based Region Importance Learning for Neural Classification of Gait States from EEG Signals

et al. 2017

View full text Add to dashboard Cite

With the development of Brain Machine Interface (BMI) systems, people with motor disabilities are able to control external devices to help them restore movement abilities. Longitudinal validation of these systems is critical not only to assess long-term performance reliability but also to investigate adaptations in electrocortical patterns due to learning to use the BMI system. In this paper, we decode the patterns of user's intended gait states (e.g., stop, walk, turn left, and turn right) from scalp electroencephalography (EEG) signals and simultaneously learn the relative importance of different brain areas by using the multiple kernel learning (MKL) algorithm. The region of importance (ROI) is identified during training the MKL for classification. The efficacy of the proposed method is validated by classifying different movement intentions from two subjects—an able-bodied and a spinal cord injury (SCI) subject. The preliminary results demonstrate that frontal and fronto-central regions are the most important regions for the tested subjects performing gait movements, which is consistent with the brain regions hypothesized to be involved in the control of lower-limb movements. However, we observed some regional changes comparing the able-bodied and the SCI subject. Moreover, in the longitudinal experiments, our findings exhibit the cortical plasticity triggered by the BMI use, as the classification accuracy and the weights for important regions—in sensor space—generally increased, as the user learned to control the exoskeleton for movement over multiple sessions.

show abstract

“…An adaptive approach was taken in (Hsu S. H. et al, 2016), who developed a optimized online recursive ICA algorithm (ORICA) using online recursive least squares (RLS) for real-time blind source separation (BSS), which is of obvious interest and use to BCI. Another interesting study was (Luu T. P. et al, 2016), who used closed-loop EEG-based BCI system for controlling the gait of a virtual avatar in VR in a BCI-VR setting. They used slow cortical potentials (0.1-3Hz) activity, achieving R values of around 0.4 for joint control of the virtual avatar on average.…”

Section: Introductionmentioning

confidence: 99%

Tracking and optimizing human performance using deep reinforcement learning in closed-loop behavioral- and neuro- feedback: a proof of concept

Martín¹,

Leech²

2017

Preprint

View full text Add to dashboard Cite

6Reinforcement learning (RL) is a general-purpose powerful machine learning 7 framework within which we can model various deterministic, non-deterministic and 8 complex environments. We applied RL to the problem of tracking and improving 9 human sustained attention during a simple sustained attention to response task 10 (SART) in a proof of concept study with two subjects, using state-of-the-art deep 11 neural network-based RL in the form of Deep Q Networks (DQNs). While others 12 have used RL in EEG settings previously, none have applied it in a neurofeedback 13 (NFB) setting, which seems a natural problem within Brain Computer Interfaces 14 (BCIs) to tackle using end-to-end RL in the form of DQNs, due to both the problem's 15 non-stationarity and the ability of RL to learn in a continuous setting. Furthermore, 16 while many have explored phasic alerting previously, learning optimal alerting in a 17 personalized way in real time is a less explored field, which we believe RL to be an 18 most suitable solution for. First, we used empirically-derived simulated data of EEG 19 and reaction times and subsequent parameter/algorithmic exploration within this 20 simulated model to pick parameters for the DQN that are more likely to be optimal for 21 the experimental setup and to explore the behavior of DQNs in this task setting. We 22 then applied the method on two subjects and show that we get different but plausible 23 results for both subjects, suggesting something about the behavior of DQNs in this 24 setting. For this experimental part, we used parameters suggested to us by the 25 simulation results. This RL-based behavioral-and neuro-feedback BCI method we 26 have developed here is input feature agnostic and allows for complex continuous 27 actions to be learned in other more complex closed-loop behavioral or neuro-feedback 28 approaches. 29

show abstract

Gait adaptation to visual kinematic perturbations using a real-time closed-loop brain–computer interface to a virtual reality avatar

Cited by 95 publications

References 49 publications

An empirical comparison of neural networks and machine learning algorithms for EEG gait decoding

An empirical comparison of neural networks and machine learning algorithms for EEG gait decoding

Multiple Kernel Based Region Importance Learning for Neural Classification of Gait States from EEG Signals

Tracking and optimizing human performance using deep reinforcement learning in closed-loop behavioral- and neuro- feedback: a proof of concept

Contact Info

Product

Resources

About