Balancing Memorization and Generalization in RNNs for High Performance Brain-Machine Interfaces

Abstract

Brain-machine interfaces (BMIs) can restore motor function to people with paralysis but are currently limited by the accuracy of real-time decoding algorithms. Recurrent neural networks (RNNs) using modern training techniques have shown promise in accurately predicting movements from neural signals but have yet to be rigorously evaluated against other decoding algorithms in a closed-loop setting. Here we compared RNNs to other neural network architectures in real-time, continuous decoding of finger movements using intracortical signals from nonhuman primates. Across one and two finger online tasks, LSTMs (a type of RNN) outperformed convolutional and transformer-based neural networks, averaging 18% higher throughput than the convolution network. On simplified tasks with a reduced movement set, RNN decoders were allowed to memorize movement patterns and matched able-bodied control. Performance gradually dropped as the number of distinct movements increased but did not go below fully continuous decoder performance. Finally, in a two-finger task where one degree-of-freedom had poor input signals, we recovered functional control using RNNs trained to act both like a movement classifier and continuous decoder. Our results suggest that RNNs can enable functional real-time BMI control by learning and generating accurate movement patterns.

A spiking neural network with continuous local learning for robust online brain machine interface

“…The performance of the proposed continuous learning algorithm for SNNs was benchmarked against several established methods: SNN trained using backpropagation-through-time (BPTT), an SNN trained using unmodified DeColle learning algorithm [39], a Kalman filter, and a LSTM decoder. The Kalman filter is often used as a standard benchmark for comparing new BMI decoders [20,43], while the LSTM was selected due to recent work demonstrating superior accuracy to other methods [11,44]. Additionally, an SNN trained with unmodified DeColle learning rules described in [39] to demonstrate the necessity of the novel continuous learning algorithm.…”

“…The BPTT SNN was included to show how the continuous learning rule affects adaptability. LSTM decoders have shown the ability to surpass the performance of Kalman filters for both offline and online decoding tasks [11,44], so it provides an additional comparison standard for the current state-of-the-art. The Kalman filter demonstrates increased performance with binned spike rates, so a sliding window of 10 time samples was used to bin the neural data before input to the Kalman filter.…”

Artificial neural network for brain-machine interface consistently produces more naturalistic finger movements than linear methods

“…In particular, nonlinear BMIs have recently enabled high-performance speech decoding in real-time ( 6 , 37 ). Recently, our group developed both a temporally-convolved feedforward neural network ( tcFNN ) which outperformed the ReFIT Kalman Filter and a recurrent neural network (RNN) which outperformed the tcFNN and a velocity Kalman Filter in a 2-degree-of-freedom (DOF) dexterous finger movement task ( 38 , 39 ).…”

“…Additionally, stochasticity (a.k.a. randomness) in ANN training approaches may lead to model instability, or variability in training, even when trained on the same data ( 39 ). Although an increasing number of studies have focused on adapting known ML architectures to neural decoding, there is a need to better understand whether ANNs will generalize and consistently converge to high-performing decoders.…”

A real-time, high-performance brain-computer interface for finger decoding and quadcopter control

“…However, after the participant grew more accustomed to the task (final 4 blocks), acquisition time for the 4D decoder dropped by an average of 0.4 s to 1.58 ± 0.06 s (a target acquisition rate of 76 ± 2 targets/min), and 100% of trials were completed. To compare this work with the previous NHP 2-finger task where throughput varied from 1.98 to 3.04 bps with a variety of decoding algorithms 23,25 , throughput for the current method was calculated as 2.64 ± 0.09 bps (see Methods for details). Table 1 summarizes statistics for the 4D decoder/task and 2D decoder/task.…”