A particle-based policy for the optimal control of Markov decision processes

Pirotta, Matteo; Manganini, Giorgio; Piroddi, Luigi; Prandini, Maria; Restelli, Marcello

doi:10.3182/20140824-6-za-1003.01987

Cited by 2 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Introduction of a particle-based representation of the control policy for MDPs with finite control space; • Extension of the PGPE to this representation, which also includes a categorical distribution for the selection of the control action; • Design of an iterative procedure for the structure selection of the particle-based policy parametrization. A preliminary version of this work has been presented in [30], and is here extended and generalized, in particular by removing the limitation that the policy be characterized statically by a fixed number of particles with pre-assigned action.…”

Section: Introductionmentioning

confidence: 99%

Policy Search for the Optimal Control of Markov Decision Processes: A Novel Particle-Based Iterative Scheme

Manganini

Pirotta

Restelli

et al. 2016

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Classical approximate dynamic programming techniques based on state-space gridding become computationally impracticable for high-dimensional problems. Policy search techniques cope with this curse of dimensionality issue by searching for the optimal control policy in a restricted parameterized policy space. We here focus on the case of discrete action space and introduce a novel policy parametrization that adopts particles to describe the map from the state space to the action space, each particle representing a region of the state space that is mapped into a certain action. The locations and actions associated with the particles describing a policy can be tuned by means of a recently introduced policy gradient method with parameter-based exploration. The task of selecting an appropriately sized set of particles is here solved through an iterative policy building scheme that adds new particles to improve the policy performance and is also capable of removing redundant particles. Experiments demonstrate the scalability of the proposed approach as the dimensionality of the state-space grows.

show abstract

Section: Introductionmentioning

confidence: 99%

Policy Search for the Optimal Control of Markov Decision Processes: A Novel Particle-Based Iterative Scheme

Manganini

Pirotta

Restelli

et al. 2016

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“…The advantages are: I) less noisy estimate both from theoretical and empirical analysis [7]; II) possibility to exploit non differentiable policies. Furthermore, PGPE has been shown to outperform standard RL gradient approaches in many complex scenarios [8], [9].…”

Section: Introductionmentioning

confidence: 99%

Following Newton direction in Policy Gradient with parameter exploration

Manganini

Pirotta

Restelli

et al. 2015

2015 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

This paper investigates the use of second-order methods to solve Markov Decision Processes (MDPs). Despite the popularity of second-order methods in optimization literature, so far little attention has been paid to the extension of such techniques to face sequential decision problems. Here we provide a model-free Reinforcement Learning method that estimates the Newton direction by sampling directly in the parameter space. In order to compute the Newton direction we provide the formulation of the Hessian of the expected return, a technique for variance reduction in the sample-based estimation and a finite sample analysis in the case of Normal distribution. Beside discussing the theoretical properties, we empirically evaluate the method on an instructional linear-quadratic regulator and on a complex dynamical quadrotor system

show abstract

A particle-based policy for the optimal control of Markov decision processes

Cited by 2 publications

References 12 publications

Policy Search for the Optimal Control of Markov Decision Processes: A Novel Particle-Based Iterative Scheme

Policy Search for the Optimal Control of Markov Decision Processes: A Novel Particle-Based Iterative Scheme

Following Newton direction in Policy Gradient with parameter exploration

Contact Info

Product

Resources

About