Quantized Attention-Gated Kernel Reinforcement Learning for Brain–Machine Interface Decoding

Wang, Fang; Wang, Yiwen; Xu, Kai; Li, Hongbao; Liao, Yuxi; Zhang, Qiaosheng; Zhang, Shaomin; Zheng, Xiaoxiang; Prı́ncipe, José C.

doi:10.1109/tnnls.2015.2493079

Cited by 38 publications

(30 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some studies have implemented this method and have achieved good decoding performance; however, most of them commonly employ supervised learning and train the decoder by mapping the recorded neural activities to some kinematic outputs, such as the real movement trajectory or the movement labels [ 25 , 26 , 27 ]. In clinical applications, the kinematic outputs may be difficult to collect, particularly for paralysis or limb amputations [ 28 , 29 , 30 , 31 ]. To address this problem, reinforcement learning (RL)-based iBMIs have been developed.…”

Section: Introductionmentioning

confidence: 99%

“…However, AGREL is sensitive to initialization and has to re-initialize numerous times to avoid becoming stuck in a poor performance, which affects the real-time capabilities and prohibits it from meeting the requirements of online decoding. Moreover, disparities between historical and new data caused by nonstationarity enlarge the state-action space and bring about more challenges to the exploration efficiency [ 29 , 30 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain–Machine Interface

Zhang

Chao

Chen

et al. 2020

Sensors

View full text Add to dashboard Cite

Background: For the nonstationarity of neural recordings in intracortical brain–machine interfaces, daily retraining in a supervised manner is always required to maintain the performance of the decoder. This problem can be improved by using a reinforcement learning (RL) based self-recalibrating decoder. However, quickly exploring new knowledge while maintaining a good performance remains a challenge in RL-based decoders. Methods: To solve this problem, we proposed an attention-gated RL-based algorithm combining transfer learning, mini-batch, and weight updating schemes to accelerate the weight updating and avoid over-fitting. The proposed algorithm was tested on intracortical neural data recorded from two monkeys to decode their reaching positions and grasping gestures. Results: The decoding results showed that our proposed algorithm achieved an approximate 20% increase in classification accuracy compared to that obtained by the non-retrained classifier and even achieved better classification accuracy than the daily retraining classifier. Moreover, compared with a conventional RL method, our algorithm improved the accuracy by approximately 10% and the online weight updating speed by approximately 70 times. Conclusions: This paper proposed a self-recalibrating decoder which achieved a good and robust decoding performance with fast weight updating and might facilitate its application in wearable device and clinical practice.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain–Machine Interface

Zhang

Chao

Chen

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…The limitation of this approach is that the large neural state-action pair gives rise to the curse of dimensionality problem leading to generalization difficulty. To overcome this problem, [17] proposed applying Attention-gated reinforcement learning (AGREL) and its variants [18], [19]. [17] reports improvement over Q-learning involving one NHP performing a center-out cursor task.…”

Section: Introductionmentioning

confidence: 99%

Lightweight Reinforcement Algorithms for autonomous, scalable intra-cortical Brain Machine Interfaces

Shaikh

Sibindi

et al. 2020

Preprint

View full text Add to dashboard Cite

Intra-cortical Brain Machine Interfaces (iBMIs) with wireless capability could scale the number of recording channels by integrating an intention decoder to reduce data rates. However, the need for frequent retraining due to neural signal non-stationarity is a big impediment. This paper presents an alternate paradigm of online reinforcement learning (RL) with a binary evaluative feedback in iBMIs to tackle this issue. This paradigm eliminates time-consuming calibration procedures. Instead, it relies on updating the model on a sequential sample-by-sample basis based on an instantaneous evaluative binary feedback signal. However, batch updates of weight in popular deep networks is very resource consuming and incompatible with constraints of an implant. In this work, using offline open-loop analysis on pre-recorded data, we show application of a simple RL algorithm - Banditron -in discrete-state iBMIs and compare it against previously reported state of the art RL algorithms – Hebbian RL, Attention gated RL, deep Q-learning. Owing to its simplistic single-layer architecture, Banditron is found to yield at least two orders of magnitude of reduction in power dissipation compared to state of the art RL algorithms. At the same time, post-hoc analysis performed on four pre-recorded experimental datasets procured from the motor cortex of two non-human primates performing joystick-based movement-related tasks indicate Banditron performing significantly better than state of the art RL algorithms by at least 5%, 10%, 7% and 7% in experiments 1, 2, 3 and 4 respectively. Furthermore, we propose a non-linear variant of Banditron, Banditron-RP, which gives an average improvement of 6%, 2% in decoding accuracy in experiments 2,4 respectively with only a moderate increase in power consumption.

show abstract

“…This framework enables a direct mapping of input spike timings into an RKHS. The kernel methods have already been applied in mapping binary input spike trains or spike counts into an RKHS to decode continuous movement trajectories or motor states (Bae, Chhatbar, Francis, Sanchez, & Príncipe, 2011;Wang et al, 2017;Zhang, Príncipe, & Wang, 2018), but they have not been developed to output discrete spike or designed to directly operate on input spike timings.…”

mentioning

confidence: 99%

Binless Kernel Machine: Modeling Spike Train Transformation for Cognitive Neural Prostheses

et al. 2020

Self Cite

View full text Add to dashboard Cite

Modeling spike train transformation among brain regions helps in designing a cognitive neural prosthesis that restores lost cognitive functions. Various methods analyze the nonlinear dynamic spike train transformation between two cortical areas with low computational eficiency. The application of a real-time neural prosthesis requires computational eficiency, performance stability, and better interpretation of the neural firing patterns that modulate target spike generation. We propose the binless kernel machine in the point-process framework to describe nonlinear dynamic spike train transformations. Our approach embeds the binless kernel to eficiently capture the feedforward dynamics of spike trains and maps the input spike timings into reproducing kernel Hilbert space (RKHS). An inhomogeneous Bernoulli process is designed to combine with a kernel logistic regression that operates on the binless kernel to generate an output spike train as a point process. Weights of the proposed model are estimated by maximizing the log likelihood of output spike trains in RKHS, which allows a global-optimal solution. To reduce computational complexity, we design a streaming-based clustering algorithm to extract typical and important spike train features. The cluster centers and their weights enable the visualization of the important input spike train patterns that motivate or inhibit output neuron firing. We test the proposed model on both synthetic data and real spike train data recorded from the dorsal premotor cortex and the primary motor cortex of a monkey performing a center-out task. Performances are evaluated by discrete-time rescaling Kolmogorov-Smirnov tests. Our model outperforms the existing methods with higher stability regardless of weight initialization and demonstrates higher eficiency in analyzing neural patterns from spike timing with less historical input (50%). Meanwhile, the typical spike train patterns selected according to weights are validated to encode output spike from the spike train of single-input neuron and the interaction of two input neurons.

show abstract

Quantized Attention-Gated Kernel Reinforcement Learning for Brain–Machine Interface Decoding

Cited by 38 publications

References 42 publications

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain–Machine Interface

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain–Machine Interface

Lightweight Reinforcement Algorithms for autonomous, scalable intra-cortical Brain Machine Interfaces

Binless Kernel Machine: Modeling Spike Train Transformation for Cognitive Neural Prostheses

Contact Info

Product

Resources

About