Learning model-free robot control by a Monte Carlo EM algorithm

Abstract

We address the problem of learning robot control by model-free reinforcement learning (RL). We adopt the probabilistic model of Vlassis and Toussaint (2009) for model-free RL, and we propose a Monte Carlo EM algorithm (MCEM) for control learning that searches directly in the space of controller parameters using information obtained from randomly generated robot trajectories. MCEM is related to, and generalizes, the PoWER algorithm of Kober and Peters (2009). In the finite-horizon case MCEM reduces precisely to PoWER, but MCEM can also handle the discounted infinite-horizon case. An interesting result is that the infinite-horizon case can be viewed as a 'randomized' version of the finite-horizon case, in the sense that the length of each sampled trajectory is a random draw from an appropriately constructed geometric distribution. We provide some preliminary experiments demonstrating the effects of fixed (PoWER) vs randomized (MCEM) horizon length in two simulated and one real robot control tasks.

Learning motor skills: from algorithms to robot experiments

“…Our novel algorithm, PoWER, is based on an expectation-maximization inspired optimization and a structured, state-dependent exploration. Our approach has already given rise to follow-up work in other contexts, for example, [Vlassis et al, 2009, Kormushev et al, 2010. Theodorou et al [2010] have shown that an algorithm very similar to PoWER can also be derived from a completely different perspective, that is, the path integral approach.…”

“…When the reward is treated as an improper probability distribution [Dayan and Hinton, 1997], safe and fast methods can be derived that are inspired by expectation-maximization. Some of these approaches have proven successful in robotics, e.g., reward-weighted regression [Peters and Schaal, 2008b], Policy Learning by Weighting Exploration with the Returns , Monte Carlo Expectation-Maximization [Vlassis et al, 2009], Cost-regularized Kernel Regression , and Policy Improvements with Path Integrals [Theodorou et al, 2010]. An overview of publications using policy search methods is presented in Table 2.2.…”

“…Applications include learning helicopter flight [Bagnell and Schneider, 2001], learning biped walk patterns , Tedrake et al, 2005, to drive a radio-controlled (RC) car as well as a jumping behavior for a robot dog jump [Kolter and Ng, 2009], as illustrated in Figure 2.3, and to balance a two wheeled robot [Vlassis et al, 2009]. Operational space control was also learned by Peters and Schaal [2008b] using locally linear controller models.…”

“…Our approach has inspired follow-up work in other contexts, for example [Vlassis et al, 2009, Kormushev et al, 2010. Theodorou et al [2010] have derived algorithms based on the path integral approach that are very similar to PoWER and have also been successfully employed for robotic tasks [Buchli et al, 2011, Tamosiunaite et al, 2011.…”

Learning the behavior model of a robot

“…Interestingly, the EM algorithm family along with Monte Carlo sampling has also been used in [65] for model-free reinforcement learning, in order to obtain directly the policy without learning the model first. In [40], the authors use this kind of technique to optimize motion of a robot in order to minimize uncertainty of localization.…”