Autonomous helicopter control using reinforcement learning policy search methods

Abstract

Abstract| Many control problems in the robotics eld can be cast as Partially Observed Markovian Decision Problems POMDPs, an optimal control formalism. Finding optimal solutions to such problems in general, however is known to be intractable. It has often been observed that in practice, simple structured controllers su ce for good sub-optimal control, and recent research in the arti cial intelligence community has focused on policy search methods as techniques for nding sub-optimal controllers when such structured controllers do exist. Traditional model-based reinforcement learning algorithms make a certainty equivalence assumption on their learned models and calculate optimal policies for a maximumlikelihood Markovian model. In this work, we consider algorithms that evaluate and synthesize controllers under distributions of Markovian models. Previous work has demonstrated that algorithms that maximize mean reward with respect to model uncertainty leads to safer and more robust controllers. We consider brie y other performance criterion that emphasize robustness and exploration in the search for controllers, and note the relation with experiment design and active learning. To v alidate the power of the approach on a robotic application we demonstrate the presented learning control algorithm by ying an autonomous helicopter. We show that the controller learned is robust and delivers good performance in this real-world domain.

Learning motor skills: from algorithms to robot experiments

“…In fact, many of the methods that scale to the most interesting tasks are model-based and often employ policy search rather than value function-based approaches , Miyamoto et al, 1996, Bagnell and Schneider, 2001, Kohl and Stone, 2004, Tedrake et al, 2005, Peters and Schaal, 2008b,c, Kober and Peters, 2008. This stands in contrast to perhaps the bulk of [Kaelbling et al, 1996, Sutton andBarto, 1998] research in the machine learning community.…”

“…The computation of the policy update is the key step here and a variety of updates have been proposed ranging from pairwise comparisons [Strens andMoore, 2001, Ng et al, 2004a] over gradient estimation using finite policy differences [Geng et al, 2006, Mitsunaga et al, 2005, Sato et al, 2002, Tedrake et al, 2005, and general stochastic optimization methods (such as Nelder-Mead [Bagnell and Schneider, 2001], cross entropy [Rubinstein and Kroese, 2004] and population-based methods [Goldberg, 1989]) to approaches coming from optimal control such as differential dynamic programming (DDP) [Atkeson, 1998] and multiple shooting approaches [Betts, 2001] as well as core reinforcement learning methods.…”

“…Applications include learning helicopter flight [Bagnell and Schneider, 2001], learning biped walk patterns , Tedrake et al, 2005, to drive a radio-controlled (RC) car as well as a jumping behavior for a robot dog jump [Kolter and Ng, 2009], as illustrated in Figure 2.3, and to balance a two wheeled robot [Vlassis et al, 2009]. Operational space control was also learned by Peters and Schaal [2008b] using locally linear controller models.…”

Autonomous Autorotation of an RC Helicopter

“…Examples range from basic upright hovering and forward flight [4,10,15,16,17] to inverted hovering [14], and even to extreme aerobatic maneuvers [1,6,5].…”

“…In particular, the model we present in this paper explicitly incorporates a model for the rotor speed dynamics, a crucial aspect of helicopter flight during autorotation. 4 Then, since it can be very difficult to specify helicopter maneuvers by hand, we use the expert demonstrations to define the autorotation task. (See also, e.g., [5], where demonstrations were used to enable a helicopter to fly high performance helicopter aerobatics.)…”