Brain-machine interface control of a robot arm using actor-critic rainforcement learning

Abstract

Here we demonstrate how a marmoset monkey can use a reinforcement learning (RL) Brain-Machine Interface (BMI) to effectively control the movements of a robot arm for a reaching task. In this work, an actor-critic RL algorithm used neural ensemble activity in the monkey's motor cortext to control the robot movements during a two-target decision task. This novel approach to decoding offers unique advantages for BMI control applications. Compared to supervised learning decoding methods, the actor-critic RL algorithm does not require an explicit set of training data to create a static control model, but rather it incrementally adapts the model parameters according to its current performance, in this case requiring only a very basic feedback signal. We show how this algorithm achieved high performance when mapping the monkey's neural states (94%) to robot actions, and only needed to experience a few trials before obtaining accurate real-time control of the robot arm. Since RL methods responsively adapt and adjust their parameters, they can provide a method to create BMIs that are robust against perturbations caused by changes in either the neural input space or the output actions they generate under different task requirements or goals.

Feature extraction and unsupervised classification of neural population reward signals for reinforcement based BMI

“…At present we have developed and used an actor critic decoding paradigm that uses an ideal feedback or feedback from the environment to control a robot arm for a two-choice task [10][11][12]. The next step is to incorporate the methods in this paper to give a processed biological signal as the feedback.…”

“…[8,9]. Using this approach, we have developed a new method of decoding that is based on actor-critic RL [10][11][12]. In this approach, the actor is driven by motor neural inputs and translates them into behavioral actions.…”

Using Reinforcement Learning to Provide Stable Brain-Machine Interface Control Despite Neural Input Reorganization

“…Rather than using supervised adaptation, we are developing a new class of neural decoders based on Reinforcement Learning (RL) [33], [34]. RL is an interactive learning method designed to allow systems to obtain reward by learning to interact with the environment, and which has adaptation built into the algorithm itself using an evaluative scalar feedback signal [35].…”

Correntropy kernel temporal differences for reinforcement learning brain machine interfaces

“…A BMI architecture based on reinforcement learning (RLBMI) was introduced in [3], and successful applications of this approach can be found in [4], [5]. In the RLBMI structure (Figure 1), the agent learns how to translate the neural states into actions based on reward values from the environment.…”