Artificial neural network for brain-machine interface consistently produces more naturalistic finger movements than linear methods

Temmar, Hisham; Willsey, Matthew S.; Costello, Joseph T.; Mender, Matthew J.; Cubillos, Luis H.; Lam, Jordan LW; Wallace, Dylan M; Kelberman, Madison M.; Patil, Parag G.; Chestek, Cynthia A.

doi:10.1101/2024.03.01.583000

Cited by 2 publications

(1 citation statement)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The second problem is the non-linear representation of finger movements - the magnitude of neural activity is non-linearly related to the finger velocities (Willsey et al 2022; Temmar et al 2024), and the activities of individual fingers sum pseudo-linearly during simultaneous finger movements (Shah et al 2023). This calls for using a non-linear model to decode attempted finger velocities from neural activity.…”

Section: Methodsmentioning

confidence: 99%

A flexible intracortical brain-computer interface for typing using finger movements

Shah,

Willsey,

Hahn

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Keyboard typing with finger movements is a versatile digital interface for users with diverse skills, needs, and preferences. Currently, such an interface does not exist for people with paralysis. We developed an intracortical brain-computer interface (BCI) for typing with attempted flexion/extension movements of three finger groups on the right hand, or both hands, and demonstrated its flexibility in two dominant typing paradigms. The first paradigm is ″point-and-click″ typing, where a BCI user selects one key at a time using continuous real-time control, allowing selection of arbitrary sequences of symbols. During cued character selection with this paradigm, a human research participant with paralysis achieved 30-40 selections per minute with nearly 90% accuracy. The second paradigm is ″keystroke″ typing, where the BCI user selects each character by a discrete movement without real-time feedback, often giving a faster speed for natural language sentences. With 90 cued characters per minute, decoding attempted finger movements and correcting errors using a language model resulted in more than 90% accuracy. Notably, both paradigms matched the state-of-the-art for BCI performance and enabled further flexibility by the simultaneous selection of multiple characters as well as efficient decoder estimation across paradigms. Overall, the high-performance interface is a step towards the wider accessibility of BCI technology by addressing unmet user needs for flexibility.

show abstract

Section: Methodsmentioning

confidence: 99%

A flexible intracortical brain-computer interface for typing using finger movements

Shah,

Willsey,

Hahn

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Exploring the trade-off between deep-learning and explainable models for brain-machine interfaces

Cubillos,

Revach,

Mender

et al. 2024

Preprint

View full text Add to dashboard Cite

People with brain or spinal cord-related paralysis often need to rely on others for basic tasks, limiting their independence. A potential solution is brain-machine interfaces (BMIs), which could allow them to voluntarily control external devices (e.g., robotic arm) by decoding brain activity to movement commands. In the past decade, deep-learning decoders have achieved state-of-the-art results in most BMI applications, ranging from speech production to finger control. However, the 'black-box' nature of deep-learning decoders could lead to unexpected behaviors, resulting in major safety concerns in real-world physical control scenarios. In these applications, explainable but lower-performing decoders, such as the Kalman filter (KF), remain the norm. In this study, we designed a BMI decoder based on KalmanNet, an extension of the KF that augments its operation with recurrent neural networks to compute the Kalman gain. This results in a varying "trust" that shifts between inputs and dynamics. We used this algorithm to predict finger movements from the brain activity of two monkeys. We compared KalmanNet results offline (pre-recorded data, n=13 days) and online (real-time predictions, n=5 days) with a simple KF and two recent deep-learning algorithms: tcFNN (non-ReFIT version) and LSTM. KalmanNet achieved comparable or better results than other deep learning models in offline and online modes, relying on the dynamical model for stopping while depending more on neural inputs for initiating movements. We further validated this mechanism by implementing a heteroscedastic KF that used the same strategy, and it also approached state-of-the-art performance while remaining in the explainable domain of standard KFs. However, we also see two downsides to KalmanNet. KalmanNet shares the limited generalization ability of existing deep-learning decoders, and its usage of the KF as an inductive bias limits its performance in the presence of unseen noise distributions. Despite this trade-off, our analysis successfully integrates traditional controls and modern deep-learning approaches to motivate high-performing yet still explainable BMI designs.

show abstract

Artificial neural network for brain-machine interface consistently produces more naturalistic finger movements than linear methods

Cited by 2 publications

References 68 publications

A flexible intracortical brain-computer interface for typing using finger movements

A flexible intracortical brain-computer interface for typing using finger movements

Exploring the trade-off between deep-learning and explainable models for brain-machine interfaces

Contact Info

Product

Resources

About