Robot motor skill coordination with EM-based Reinforcement Learning

Kormushev, Petar; Calinon, Sylvain; Caldwell, Darwin G.

doi:10.1109/iros.2010.5649089

Cited by 216 publications

(189 citation statements)

References 24 publications

Supporting

Mentioning

187

Contrasting

Unclassified

Order By: Relevance

“…Our novel algorithm, PoWER, is based on an expectation-maximization inspired optimization and a structured, state-dependent exploration. Our approach has already given rise to follow-up work in other contexts, for example, [Vlassis et al, 2009, Kormushev et al, 2010. Theodorou et al [2010] have shown that an algorithm very similar to PoWER can also be derived from a completely different perspective, that is, the path integral approach.…”

Section: Policy Learning By Weighting Exploration With the Returns (Pmentioning

confidence: 94%

“…Our approach has inspired follow-up work in other contexts, for example [Vlassis et al, 2009, Kormushev et al, 2010. Theodorou et al [2010] have derived algorithms based on the path integral approach that are very similar to PoWER and have also been successfully employed for robotic tasks [Buchli et al, 2011, Tamosiunaite et al, 2011.…”

Section: Policy Learning By Weighting Exploration With the Returns (Pmentioning

confidence: 99%

See 1 more Smart Citation

Learning motor skills: from algorithms to robot experiments

Kober

Peters

2014

It - Information Technology

View full text Add to dashboard Cite

Die Veröffentlichung steht unter folgender Creative Commons Lizenz: Namensnennung -Keine kommerzielle Nutzung -Keine Bearbeitung 2.0 Deutschland http://creativecommons.org/licenses/by-nc-nd/2.0/de/ Abstract Ever since the word "robot" was introduced to the English language by KarelČapek's play "Rossum's Universal Robots" in 1921, robots have been expected to become part of our daily lives. In recent years, robots such as autonomous vacuum cleaners, lawn mowers, and window cleaners, as well as a huge number of toys have been made commercially available. However, a lot of additional research is required to turn robots into versatile household helpers and companions. One of the many challenges is that robots are still very specialized and cannot easily adapt to changing environments and requirements. Since the 1960s, scientists attempt to provide robots with more autonomy, adaptability, and intelligence. Research in this field is still very active but has shifted focus from reasoning based methods towards statistical machine learning. Both navigation (i.e., moving in unknown or changing environments) and motor control (i.e., coordinating movements to perform skilled actions) are important sub-tasks.In this thesis, we will discuss approaches that allow robots to learn motor skills. We mainly consider tasks that need to take into account the dynamic behavior of the robot and its environment, where a kinematic movement plan is not sufficient. The presented tasks correspond to sports and games but the presented techniques will also be applicable to more mundane household tasks. Motor skills can often be represented by motor primitives. Such motor primitives encode elemental motions which can be generalized, sequenced, and combined to achieve more complex tasks. For example, a forehand and a backhand could be seen as two different motor primitives of playing table tennis. We show how motor primitives can be employed to learn motor skills on three different levels. First, we discuss how a single motor skill, represented by a motor primitive, can be learned using reinforcement learning. Second, we show how such learned motor primitives can be generalized to new situations. Finally, we present first steps towards using motor primitives in a hierarchical setting and how several motor primitives can be combined to achieve more complex tasks.To date, there have been a number of successful applications of learning motor primitives employing imitation learning. However, many interesting motor learning problems are high-dimensional reinforcement learning problems which are often beyond the reach of current reinforcement learning methods. We review research on reinforcement learning applied to robotics and point out key challenges and important strategies to render reinforcement learning tractable. Based on these insights, we introduce novel learning approaches both for single and generalized motor skills.For learning single motor skills, we study parametrized policy search methods and introduce a framework of reward-weighted imi...

show abstract

Section: Policy Learning By Weighting Exploration With the Returns (Pmentioning

confidence: 94%

Section: Policy Learning By Weighting Exploration With the Returns (Pmentioning

confidence: 99%

Learning motor skills: from algorithms to robot experiments

Kober

Peters

2014

It - Information Technology

View full text Add to dashboard Cite

show abstract

“…They have been used successfully to solve many complex tasks, including the 'Ball-in-the-Cup' game [4], Ball-Throwing [5], Pancake-Flipping [6] and bipedal gait generation [7]. The original motivation for using MPs is to compose complex behavior out of simpler building blocks of movement, or movement primitives.…”

Section: Introductionmentioning

confidence: 99%

“…Such representations use time-or phase-dependent policies [1], [6], [9]. For example, the widely used dynamic movement primitive (DMP) approach [1] uses a phase signal to implement execution speed adaptation for the movement.…”

Section: Introductionmentioning

confidence: 99%

“…The DMP framework is well-established in robotics as they are straightforward to obtain from imitation [1] and reinforcement learning [10], can be used for both rhythmic and stroke-based movements, the movement can be generalized to different final-positions, and the movement speed can be adjusted by a temporal scaling parameter. The original DMP framework has been extended in [3] to generalize also to different final velocities and in [6] to use a timevarying spring-damper system which effectively modulates the stiffness of the executed movement.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A probabilistic approach to robot trajectory generation

Paraschos

Neumann

Peters

2013

2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids)

View full text Add to dashboard Cite

Abstract-Motor Primitives (MPs) are a promising approach for the data-driven acquisition as well as for the modular and re-usable generation of movements. However, a modular control architecture with MPs is only effective if the MPs support co-activation as well as continuously blending the activation from one MP to the next. In addition, we need efficient mechanisms to adapt a MP to the current situation. Common approaches to movement primitives lack such capabilities or their implementation is based on heuristics. We present a probabilistic movement primitive approach that overcomes the limitations of existing approaches. We encode a primitive as a probability distribution over trajectories. The representation as distribution has several beneficial properties. It allows encoding a time-varying variance profile. Most importantly, it allows performing new operations -a product of distributions for the co-activation of MPs conditioning for generalizing the MP to different desired targets. We derive a feedback controller that reproduces a given trajectory distribution in closed form. We compare our approach to the existing state-of-the art and present real robot results for learning from demonstration.

show abstract

Model‐based vs data‐driven adaptive control: An overview

Benosman

2018

Adaptive Control & Signal

View full text Add to dashboard Cite

In this paper, we present an overview of adaptive control by contrasting model-based approaches with data-driven approaches. Indeed, we propose to classify adaptive controllers into two main subfields, namely, model-based adaptive control and data-driven adaptive control. In each subfield, we cite monographs, survey papers, and recent research papers published in the last few years. We also include a few simple examples to illustrate some general concepts in each subfield.

show abstract

Robot motor skill coordination with EM-based Reinforcement Learning

Cited by 216 publications

References 24 publications

Learning motor skills: from algorithms to robot experiments

Learning motor skills: from algorithms to robot experiments

A probabilistic approach to robot trajectory generation

Model‐based vs data‐driven adaptive control: An overview

Contact Info

Product

Resources

About