Reinforcement learning of motor skills using Policy Search and human corrective advice

Abstract

Robot learning problems are limited by physical constraints, which make learning successful policies for complex motor skills on real systems unfeasible. Some reinforcement learning methods, like Policy Search, offer stable convergence toward locally optimal solutions, whereas interactive machine learning or learning-from-demonstration methods allow fast transfer of human knowledge to the agents. However, most methods require expert demonstrations. In this work, we propose the use of human corrective advice in the actions domain for learning motor trajectories. Additionally, we combine this human feedback with reward functions in a Policy Search learning scheme. The use of both sources of information speeds up the learning process, since the intuitive knowledge of the human teacher can be easily transferred to the agent, while the Policy Search method with the cost/reward function take over for supervising the process and reducing the influence of occasional wrong human corrections. This interactive approach has been validated for learning movement primitives with simulated arms with several degrees of freedom in reaching via-point movements, and also using real robots in such tasks as “writing characters” and the ball-in-a-cup game. Compared with standard reinforcement learning without human advice, the results show that the proposed method not only converges to higher rewards when learning movement primitives, but also that the learning is sped up by a factor of 4–40 times, depending on the task.

Reinforcement Learning With Human Advice: A Survey

“…Consequently, many advice-taking systems combine different learning modalities in order to balance between autonomy and control. For example, RL can be augmented with evaluative feedback (Judah et al, 2010 ; Sridharan, 2011 ; Knox and Stone, 2012b ), corrective feedback (Celemin et al, 2019 ), instructions (Maclin and Shavlik, 1996 ; Kuhlmann et al, 2004 ; Rosenstein et al, 2004 ; Pradyot et al, 2012b ), instructions and evaluative feedback (Najar et al, 2020b ), demonstrations (Taylor et al, 2011 ; Subramanian et al, 2016 ), demonstrations and evaluative feedback (Leon et al, 2011 ), or demonstrations, evaluative feedback, and instructions (Tenorio-Gonzalez et al, 2010 ). Demonstrations can be augmented with corrective feedback (Chernova and Veloso, 2009 ; Argall et al, 2011 ), instructions (Rybski et al, 2007 ), instructions and feedback, both evaluative and corrective (Nicolescu and Mataric, 2003 ), or with prior RL (Syed and Schapire, 2007 ).…”

Multi-Channel Interactive Reinforcement Learning for Sequential Tasks

“…Another line of work is to consider human prior knowledge of task decomposition to achieve a form of curriculum learning for more complex tasks (Wang et al, 2020 ). Human input to RL has also been used in combination with policy search methods and to improve robot skills on a trajectory level (Celemin and Ruiz-del Solar, 2016 , 2019 ; Celemin et al, 2019 ). This is also very relevant for robotic applications, however, it should be noted that in this paper we focus only on the sequencing of skills as high-level actions.…”

Reinforcement learning with human advice: a survey

“…For example, RL can be augmented with evaluative feedback [51,106,60], corrective feedback [20], instructions [79,65,102,99], instructions and evaluative feedback [90], demonstrations [112,109], demonstrations and evaluative feedback [66], or demonstrations, evaluative feedback and instructions [114]. Demonstrations can be augmented with corrective feedback [25,6], instructions [103], instructions and feedback, both evaluative and corrective [95], or with prior Reinforcement Learning [111].…”