Reinforcement Learning with Kernel Recursive Least-Squares Support Vector Machine

Abstract

Abstract-A reinforcement learning system based on the kernel recursive least-squares algorithm for continuous state-space is proposed in this paper. A kernel recursive least-squares-support vector machine is used to realized a mapping from state-action pair to Q-value function. An online sparsification process that permits the addition of training sample into the Q-function approximation only if it is approximately linearly independent of the preceding training samples. Simulation result of two-link robot manipulator show that the proposed method has high learning efficiency -better accuracy measured in terms of mean square error, and lesser computation time compare to the least-squares support vector machine.Index Terms-Kernel methods, least-squares support vector machine, recursive least squares, reinforcement learning. I. INTRODUCTIONSupport vector machine (SVM), which is based on Vapnik's structural risk minimization (SRM) [1], has become one of the most popular methods in solving classification and regression problems. Conventional SVMs have properties of global optimization, and good adaptability. However, the optimal solutions are obtained by solving standard quadratic programming which results in high computational cost. In order to reduce the computational cost, a least square support vector (LS-SVM) was proposed in [2] by converting inequality constraints to linear equations. LS-SVM has been successfully applied to reinforcement learning (RL) problems [3]. A RL problem is converted into a regression problem, wherein the observed states and actions are considered as inputs and Q-value functions as output. All the training samples may be support vectors in LS-SVM, and thus the support vectors are no longer sparse. It may lead to poor generalization. In addition, with the number of the input training pairs increasing, the number of equations will increase which may result in higher computational cost.Our focus in this paper is on reinforcement learning problems. The objective in hand is to explore the use of a support vector machine with sparse support vectors, low computational cost and satisfactory accuracy. The kernel recursive least-squares (KRLS) -SVM [4], is a strong candidate to achieve this objective. We develop a KRLS-SVM algorithm for reinforcement learning and demonstrate its potential through case study -two-link robot manipulator. The mean square error accuracy, computational Manuscript received

Policy Iteration Based Optimal Control for Partially Unknown Nonlinear Systems via Semi-parametric Regression Model

“…Remark 2: Note that although the formulas (17) and (19) have the same solution for optimal control problems, the formula (17) does not contain the knowledge of the internal dynamics ) (x f , which needs to be known explicitly in (19).…”

“…The learning result is relative to an initial value and it is difficult to converge to a unique optimal policy [18]. In addition, as a typical non-parametric kernel method, support vector machine (SVM), which is based on Vapnik's structural risk minimization (SRM) [19], has perfect generalization property and can be able to overcome the existing weakness in parametric function approximator. But it will decrease the interpretative capability of the model, when the large amount of information is provided by experience.…”

Data‐Driven Adaptive Critic Approach for Nonlinear Optimal Control via Least Squares Support Vector Machine

“…Besides, the optimum solutions are obtained by solving the standard quadratic programming (QP) problem, which limits its real-time applications [29]. Based on standard SVM, the LS-SVM was put forward and has been applied extensively since [30][31][32]. The difference between the LS-SVM and the standard SVM is that the constraint condition is equalities rather than inequalities, thus converting the QP problem into a linear equation groups problem.…”