Reinforcement learning via kernel temporal difference

Abstract

This paper introduces a kernel adaptive filter implemented with stochastic gradient on temporal differences, kernel Temporal Difference (TD)(λ), to estimate the state-action value function in reinforcement learning. The case λ=0 will be studied in this paper. Experimental results show the method's applicability for learning motor state decoding during a center-out reaching task performed by a monkey. The results are compared to the implementation of a time delay neural network (TDNN) trained with backpropagation of the temporal difference error. From the experiments, it is observed that kernel TD(0) allows faster convergence and a better solution than the neural network.

A new method of concurrently visualizing states, values, and actions in reinforcement based brain machine interfaces

“…Typically in RL, temporal difference algorithms approximate the value function using a parametrized family of functions. A nonparametric variant, KTD(λ) [3], [5], is obtained by approximating the value function using a function f ∈ H, where H is a reproducing kernel Hilbert space (RKHS) with reproducing kernel κ(·, ·), such that f (x(t)) = f, φ(x(t)) = f, κ(x(t), ·) , ∀f ∈ H with a mapping function φ(x(t)) : X → H and a positive definite function κ : X × X → R. The KTD(λ) update rule is…”

“…A BMI architecture based on reinforcement learning (RLBMI) is introduced in [1]; successful applications of this approach can be found in [2], [3], [4]. The key idea of RLBMI is co-adaptation between two intelligent systems: the BMI decoder in the agent and the BMI user in the environment ( Figure 1).…”

“…For the BMI decoder we chose Q-learning via kernel Temporal Difference (KTD)(λ) [3], [5]. The behavior task is a reaching task using a robotic arm, in which the decoder controls the robot arm's action direction by predicting the monkey's intent based on its neuronal activity.…”

Stochastic kernel temporal difference for reinforcement learning

“…In this RLBMI framework, an agent learns how to transfer the neural states into actions based on predefined reward values from the environment. The agent must interpret the subject's brain activity correctly to facilitate the rewards [18,19]. A female bonnet macaque is trained for a center-out reaching task allowing 8 action directions.…”

Correntropy kernel temporal differences for reinforcement learning brain machine interfaces

“…A BMI architecture based on reinforcement learning (RLBMI) was introduced in [3], and successful applications of this approach can be found in [4], [5]. In the RLBMI structure (Figure 1), the agent learns how to translate the neural states into actions based on reward values from the environment.…”