Model-based policy gradients with parameter-based exploration by least-squares conditional density estimation

Tangkaratt, Voot; Mori, Shunsuke; Zhao, Tingting; Morimoto, Jun; Sugiyama, Masashi

doi:10.1016/j.neunet.2014.06.006

Cited by 15 publications

(9 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both the model-based PGPE [30] and the PILCO [13] algorithm use gradient-based policy updates. Rather than using Monte Carlo sampling, as in model-based PGPE, PILCO performs deterministic approximate inference by explicitly incorporating the model uncertainty into long-term predictions.…”

Section: Related Workmentioning

confidence: 99%

Black-box data-efficient policy search for robotics

Chatzilygeroudis

Rama

Kaushik

et al. 2017

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

108

View full text Add to dashboard Cite

The most data-efficient algorithms for reinforcement learning (RL) in robotics are based on uncertain dynamical models: after each episode, they first learn a dynamical model of the robot, then they use an optimization algorithm to find a policy that maximizes the expected return given the model and its uncertainties. It is often believed that this optimization can be tractable only if analytical, gradient-based algorithms are used; however, these algorithms require using specific families of reward functions and policies, which greatly limits the flexibility of the overall approach. In this paper, we introduce a novel model-based RL algorithm, called Black-DROPS (Black-box Data-efficient RObot Policy Search) that:(1) does not impose any constraint on the reward function or the policy (they are treated as black-boxes), (2) is as dataefficient as the state-of-the-art algorithm for data-efficient RL in robotics, and (3) is as fast (or faster) than analytical approaches when several cores are available. The key idea is to replace the gradient-based optimization algorithm with a parallel, blackbox algorithm that takes into account the model uncertainties. We demonstrate the performance of our new algorithm on two standard control benchmark problems (in simulation) and a low-cost robotic manipulator (with a real robot).

show abstract

Section: Related Workmentioning

confidence: 99%

Black-box data-efficient policy search for robotics

Chatzilygeroudis

Rama

Kaushik

et al. 2017

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

108

View full text Add to dashboard Cite

show abstract

“…There exist many approaches to learn the models f and r (for model-based policy search) in the literature [47,113,179]. Most algorithms assume a known reward function; otherwise they usually use the same technique to learn both models.…”

Section: Model Learningmentioning

confidence: 99%

A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials

et al. 2020

View full text Add to dashboard Cite

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word "big-data", we refer to this challenge as "micro-data reinforcement learning". We show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based policy search), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots, designing generic priors, and optimizing the computing time.1 In some rare cases, a process can be "optimally efficient". 2 It is challenging to put a precise limit for "micro-data learning" as each domain has different experimental constraints, this is why we will refer in this article to "a few minutes" or a "a few trials". The commonly used word "big-data" has a similar "fuzzy" limit that depends on the exact domain. 3 Planning-based and model-predictive control [59] methods do not search for policy parameters, this is why they do not fit into the scope of this paper. Although trajectory-based policies and planning-based methods share the same goal, they usually search in a different space: planning algorithms search in the state-action space (e.g., joint positions/velocities), whereas policy methods will search for the optimal parameters of the policy, which can encode a Chatzilygeroudis, Vassiliades, Stulp, Calinon and Mouret arXiv | 1 arXiv:1807.02303v4 [cs.RO] 31 May 2019 Chatzilygeroudis, Vassiliades, Stulp, Calinon and Mouret arXiv | 2This is basically sampling the distribution over trajectories, P (τ |θ), which is feasible since the sampling is performed with the models. When applying the same policy (i.e., a policy with the same parameters θ), the trajectories τ (and consequently r)Chatzilygeroudis, Vassiliades, Stulp, Calinon and Mouret arXiv | 9

show abstract

“…Policy Search with MBRL The standard approach consists in using a maximum likelihood estimation of the environment dynamics to perform simulations (or imaginary rollouts) through which a policy can be improved without further or with limited interactions with the environment (Deisenroth et al 2013). This approach has taken different forms, with the use of tabular models (Wang and Dietterich 2003), least-squares density estimation techniques (Tangkaratt et al 2014) or, more recently, combinations of variational generative models and recurrent neural networks employed in world models based on mixture density networks (Ha and Schmidhuber 2018). Several methods incorporate the model uncertainty into policy updates, by using Gaussian processes and moment matching approximations (Deisenroth and Rasmussen 2011), Bayesian neural networks (Gal, McAllister, and Rasmussen 2016) or ensembles of forward models (Chua et al 2018;Kurutach et al 2018;Janner et al 2019;Buckman et al 2018).…”

Section: Related Workmentioning

confidence: 99%

Gradient-Aware Model-Based Policy Search

D’Oro

Metelli

Tirinzoni

et al. 2020

AAAI

View full text Add to dashboard Cite

Traditional model-based reinforcement learning approaches learn a model of the environment dynamics without explicitly considering how it will be used by the agent. In the presence of misspecified model classes, this can lead to poor estimates, as some relevant available information is ignored. In this paper, we introduce a novel model-based policy search approach that exploits the knowledge of the current agent policy to learn an approximate transition model, focusing on the portions of the environment that are most relevant for policy improvement. We leverage a weighting scheme, derived from the minimization of the error on the model-based policy gradient estimator, in order to define a suitable objective function that is optimized for learning the approximate transition model. Then, we integrate this procedure into a batch policy improvement algorithm, named Gradient-Aware Model-based Policy Search (GAMPS), which iteratively learns a transition model and uses it, together with the collected trajectories, to compute the new policy parameters. Finally, we empirically validate GAMPS on benchmark domains analyzing and discussing its properties.

show abstract

Model-based policy gradients with parameter-based exploration by least-squares conditional density estimation

Cited by 15 publications

References 26 publications

Black-box data-efficient policy search for robotics

Black-box data-efficient policy search for robotics

A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials

Gradient-Aware Model-Based Policy Search

Contact Info

Product

Resources

About