Value Function Approximation on Non-Linear Manifolds for Robot Motor Control

Sugiyama, Masashi; Hachiya, Hirotaka; Towell,; Vijayakumar, Sethu

doi:10.1109/robot.2007.363573

Cited by 10 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we apply our method to three benchmark (simulated) robot motor control problems: (a) pendulum swing-up with limited torque (Doya, 2000), (b) robot arm control with obstacles (Sugiyama et al, 2007), and (c) multi-degrees of freedom (DOF) redundant arm reaching task (Theodorou et al, 2010). Figure 1 illustrates these problems.…”

Section: Simulation Resultsmentioning

confidence: 99%

Latent Kullback Leibler Control for Continuous-State Systems using Probabilistic Graphical Models

Matsubara¹,

Gómez²,

Kappen³

2014

Preprint

View full text Add to dashboard Cite

Kullback Leibler (KL) control problems allow for efficient computation of optimal control by solving a principal eigenvector problem. However, direct applicability of such framework to continuous state-action systems is limited. In this paper, we propose to embed a KL control problem in a probabilistic graphical model where observed variables correspond to the continuous (possibly high-dimensional) state of the system and latent variables correspond to a discrete (low-dimensional) representation of the state amenable for KL control computation. We present two examples of this approach. The first one uses standard hidden Markov models (HMMs) and computes exact optimal control, but is only applicable to lowdimensional systems. The second one uses factorial HMMs, it is scalable to higher dimensional problems, but control computation is approximate. We illustrate both examples in several robot motor control tasks.

show abstract

Section: Simulation Resultsmentioning

confidence: 99%

Latent Kullback Leibler Control for Continuous-State Systems using Probabilistic Graphical Models

Matsubara¹,

Gómez²,

Kappen³

2014

Preprint

View full text Add to dashboard Cite

show abstract

“…Tobias and Daniel proposed a LSTD approach based on SVMs [96]. Several researchers have investigated designing specialized kernels that exploit manifold structure in the state space [90,91,59,60,10,9,87]. This work represents exciting progress; however, the field of kernel-based ADP has developed only recently, and there remain numerous possibilities that are yet unexplored.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Investigation of manifold-based kernels, and their relationship to n-stage BRE A number of researchers have proposed using kernels that exploit manifold structure on the state space as a means of devising cost approximation algorithms [90,91,59,60,10,9,87]. We believe that these kernels are particularly appropriate for use in our BRE algorithms, and propose to test these manifold-based kernels in several BRE test problems.…”

Section: Further Bre Algorithm Development/extensionmentioning

confidence: 99%

Approximate dynamic programming using Bellman residual elimination and Gaussian process regression

Bethke

How

2009

2009 American Control Conference

View full text Add to dashboard Cite

The overarching goal of the thesis is to devise new strategies for multi-agent planning and control problems, especially in the case where the agents are subject to random failures, maintenance needs, or other health management concerns, or in cases where the system model is not perfectly known. We argue that dynamic programming techniques, in particular Markov Decision Processes (MDPs), are a natural framework for addressing these planning problems, and present an MDP problem formulation for a persistent surveillance mission that incorporates stochastic fuel usage dynamics and the possibility for randomly-occurring failures into the planning process. We show that this problem formulation and its optimal policy lead to good mission performance in a number of realworld scenarios. Furthermore, an on-line, adaptive solution framework is developed that allows the planning system to improve its performance over time, even in the case where the true system model is uncertain or time-varying. Motivated by the difficulty of solving the persistent mission problem exactly when the number of agents becomes large, we then develop a new family of approximate dynamic programming algorithms, called Bellman Residual Elimination (BRE) methods, which can be employed to approximately solve large-scale MDPs. We analyze these methods and prove a number of desirable theoretical properties about them, including reduction to exact policy iteration under certain conditions. Finally, we apply these BRE methods to large-scale persistent surveillance problems and show that they yield good performance, and furthermore, that they can be successfully integrated into the adaptive planning framework.2

show abstract

“…The current paper is a complete version of our earlier manuscript (Sugiyama et al 2007). The major differences are that we included more technical details of the proposed method in Sect.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in simulated robot arm control and Khepera robot navigation.The current paper is a complete version of our earlier manuscript (Sugiyama et al 2007). The major differences are that we included more technical details of the proposed method in Sect.…”

mentioning

confidence: 99%

Geodesic Gaussian kernels for value function approximation

et al. 2008

Self Cite

View full text Add to dashboard Cite

The least-squares policy iteration approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in simulated robot arm control and Khepera robot navigation.The current paper is a complete version of our earlier manuscript (Sugiyama et al. 2007). The major differences are that we included more technical details of the proposed method in Sect. 3, discussions on the relation to related methods in Sect. 4, and the application to map building in Sect. 6. A demo movie of the proposed method applied in simulated robot arm control and Khepera robot navigation is available from

show abstract

Value Function Approximation on Non-Linear Manifolds for Robot Motor Control

Cited by 10 publications

References 19 publications

Latent Kullback Leibler Control for Continuous-State Systems using Probabilistic Graphical Models

Latent Kullback Leibler Control for Continuous-State Systems using Probabilistic Graphical Models

Approximate dynamic programming using Bellman residual elimination and Gaussian process regression

Geodesic Gaussian kernels for value function approximation

Contact Info

Product

Resources

About