Using Reinforcement Learning to Provide Stable Brain-Machine Interface Control Despite Neural Input Reorganization

Abstract

Brain-machine interface (BMI) systems give users direct neural control of robotic, communication, or functional electrical stimulation systems. As BMI systems begin transitioning from laboratory settings into activities of daily living, an important goal is to develop neural decoding algorithms that can be calibrated with a minimal burden on the user, provide stable control for long periods of time, and can be responsive to fluctuations in the decoder’s neural input space (e.g. neurons appearing or being lost amongst electrode recordings). These are significant challenges for static neural decoding algorithms that assume stationary input/output relationships. Here we use an actor-critic reinforcement learning architecture to provide an adaptive BMI controller that can successfully adapt to dramatic neural reorganizations, can maintain its performance over long time periods, and which does not require the user to produce specific kinetic or kinematic activities to calibrate the BMI. Two marmoset monkeys used the Reinforcement Learning BMI (RLBMI) to successfully control a robotic arm during a two-target reaching task. The RLBMI was initialized using random initial conditions, and it quickly learned to control the robot from brain states using only a binary evaluative feedback regarding whether previously chosen robot actions were good or bad. The RLBMI was able to maintain control over the system throughout sessions spanning multiple weeks. Furthermore, the RLBMI was able to quickly adapt and maintain control of the robot despite dramatic perturbations to the neural inputs, including a series of tests in which the neuron input space was deliberately halved or doubled.

Making brain–machine interfaces robust to future neural variability

“…A number of complementary lines of research are aimed at making BMIs more robust, including improving sensors to record from more neurons more reliably (for example, ref. 44); decoding multiunit spikes103045 or local field potentials313246 that appear to be more stable control signals than single-unit activity; and using adaptive decoders that update their parameters to follow changing neural-to-kinematic mappings4102021222324252627282947. Here we present the MRNN as a proof-of-principle of a novel approach: build a fixed decoder whose architecture allows it to be inherently robust to recording condition changes based on the assumption that novel conditions have some similarity to previously encountered conditions.…”

“…The clinical viability of BMIs would be much improved by making decoders robust to recording condition changes1819, and several recent studies have focused on this problem (for example, refs 4, 10, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29). We can broadly divide the conditions that a BMI will encounter into one of two types: (1) conditions that are completely different from what has been previously encountered; and (2) conditions that share some commonality with ones previously encountered.…”

Augmenting intracortical brain-machine interface with neurally driven error detectors

“…Chronic intracortical electrode signals degrade with time, which decreases BMI performance and success rates [92–95]. To date, efforts to rescue performance have focused on designing new kinematic decoders [56–59,63,65,96,97]. The error detection methods introduced here provide an alternative approach to increase effective success rates, thus rescuing BMI performance and improving the user experience.…”

The cortical motor system of the marmoset monkey (Callithrix jacchus)

“…Considering the increasing use of marmosets in studies of motor function and dysfunction (Marshall and Ridley, 2003;Fouad et al, 2004;Virley et al, 2004;Freret et al, 2008;Yamane et al, 2010;Konomi et al, 2012;Maggi et al, 2014;Pohlmeyer et al, 2014), a detailed examination of the circuits subserving motor control is timely.…”