Simultaneous on-line Discovery and Improvement of Robotic Skill options

Abstract

Abstract-The regularity of everyday tasks enables us to reuse existing solutions for task variations. For instance, most door-handles require the same basic skill (reach, grasp, turn, pull), but small adaptations of the basic skill are required to adapt to the variations that exist (e.g. levers vs. knobs). We introduce the algorithm "Simultaneous On-line Discovery and Improvement of Robotic Skills" (SODIRS) that is able to autonomously discover and optimize skill options for such task variations. We formalize the problem in a reinforcement learning context, and use the PI BB algorithm [2] to continually optimize skills with respect to a cost function. SODIRS discovers new subskills, or "skill options", by clustering the costs of trials, and determining whether perceptual features are able to predict which cluster a trial will belong to. This enables SODIRS to build a decision tree, in which the leaves contain skill options for task variations. We demonstrate SODIRS' performance in simulation, as well as on a Meka humanoid robot performing the ball-in-cup task.

Proximodistal exploration in motor learning as an emergent property of optimization

“…Baranes and Oudeyer (Baranes & Oudeyer, ) have studied the efficiency of combining stochastic optimization to reach goals with maturational mechanisms which progressively grow the limits within which stochastic optimization can physically explore, showing an increase in efficiency from a machine learning point of view. Several works have shown how human demonstration of movements could bootstrap this optimization process (e.g., Stulp, Herlant, Hoarau, & Raiola, ), or how humans can progressively shape subparts of the movements to complement autonomous exploration (Chernova & Thomaz, ). Finally, exploration in infants is also highly driven by mechanisms of intrinsic motivation (also called curiosity), where instead of trying to reach a goal imposed by social peers or the experimenter (as in the model presented in this paper), they use intrinsic criteria such as information gain or surprise to set their own goals and choose how to practice these self‐selected goals (Gottlieb, Oudeyer, Lopes, & Baranes, ; Moulin‐Frier, Nguyen, & Oudeyer, ).…”

“…Baranes and Oudeyer (Baranes & Oudeyer, 2011) have studied the efficiency of combining stochastic optimization to reach goals with maturational mechanisms which progressively grow the limits within which stochastic optimization can physically explore, showing an increase in efficiency from a machine learning point of view. Several works have shown how human demonstration of movements could bootstrap this optimization process (e.g., Stulp, Herlant, Hoarau, & Raiola, 2014), or how humans can progressively shape subparts of the movements to complement autonomous exploration (Chernova & Thomaz, 2014).…”

Learning Legible Motion from Human–Robot Interactions

“…Direct policy search is a form of reinforcement learning in which the search for the optimal policy is done directly in the space of the parameters θ of a parameterized policy π θ , rather than using a value function. The specific algorithm we use is PI BB (Policy Improvement through Black-Box optimization [22]). Since any model-free direct policy search algorithm could be used to implement this optimization (e.g.…”

“…Despite its simplicity, PI BB is able to learn robot skills efficiently and robustly [22]. Alternatively, algorithms such as PIˆ2, PoWER, NES, PGPE, or CMA-ES could be used, see [23,11] for an overview and comparisons.…”

Modular active curiosity-driven discovery of tool use

“…Next to approaches that have considered finite sets of parameterized problems [4], [12], other approaches [7], [8], [9], [10], [13] have considered the challenge of autonomous exploration and learning of continuous fields of parameterized problems (e.g. discovering and learning all the feasible displacements of objects and their motor solutions).…”