Gaussian process dynamic programming

Abstract

a b s t r a c tReinforcement learning (RL) and optimal control of systems with continuous states and actions require approximation techniques in most interesting cases. In this article, we introduce Gaussian process dynamic programming (GPDP), an approximate value function-based RL algorithm. We consider both a classic optimal control problem, where problem-specific prior knowledge is available, and a classic RL problem, where only very general priors can be used. For the classic optimal control problem, GPDP models the unknown value functions with Gaussian processes and generalizes dynamic programming to continuous-valued states and actions. For the RL problem, GPDP starts from a given initial state and explores the state space using Bayesian active learning. To design a fast learner, available data have to be used efficiently. Hence, we propose to learn probabilistic models of the a priori unknown transition dynamics and the value functions on the fly. In both cases, we successfully apply the resulting continuous-valued controllers to the under-actuated pendulum swing up and analyze the performances of the suggested algorithms. It turns out that GPDP uses data very efficiently and can be applied to problems, where classic dynamic programming would be cumbersome.

Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning

“…Other algorithms that use GP dynamics models in an RL setup were proposed in [20,8]. In [20,8], value function models have to be maintained, which becomes difficult in higherdimensional state spaces.…”

“…In [20,8], value function models have to be maintained, which becomes difficult in higherdimensional state spaces. Although the approaches in [20,8] do long-term planning for finding a policy, they cannot directly deal with constraints in the state space (e.g., obstacles).…”

Sequential Feature Selection for Classification

“…FQI uses a batch-trained function approximator (FA) as action-value function. Various types of non-linear function approximators have been successfully used with FQI, e.g., Neural Networks [12], Gaussian Processes [2], and others [9]. In this paper, we will use Locally Weighted Projection Regression (LWPR) [15] as the value function approximator of choice, as it is a fast robust online method that can handle large amounts of data.…”

Efficient Non-Parametric Surface Representations Using Active Sampling for Push Broom Laser Data

“…Within this field, active subsampling strategies have been used to select information-rich data through use of information theoretic criteria [8], [3], [25]. Our work is particularly similar to [25], but we exploit the time-sequential nature of laser data (see Section IV-B) to form an exact and inexpensive predictive distribution for use in our decision criterion.…”