Model-free reinforcement learning from expert demonstrations: a survey

Abstract

Reinforcement learning from expert demonstrations (RLED) is the intersection of imitation learning with reinforcement learning that seeks to take advantage of these two learning approaches. RLED uses demonstration trajectories to improve sample efficiency in high-dimensional spaces. RLED is a new promising approach to behavioral learning through demonstrations from an expert teacher. RLED considers two possible knowledge sources to guide the reinforcement learning process: prior knowledge and online knowledge.This survey focuses on novel methods for model-free reinforcement learning guided through demonstrations, commonly but not necessarily provided by humans. The methods are analyzed and classified according to the impact of the demonstrations. Challenges, applications, and promising approaches to improve the discussed methods are also discussed.

Cost Inference of Discrete-time Linear Quadratic Control Policies using Human-Behaviour Learning

“…Therefore, it is necessary to extract the hidden cost of expert's policies to infer the definition of the task to new systems and environments maintaining the expert's desired performance. However, the cost inference becomes a hard problem since the aim of ADP/RL is not to imitate performance, but use experience to facilitate learning and improve the performance of the controller [14], [15].…”

Dexterous Manipulation for Multi-Fingered Robotic Hands With Reinforcement Learning: A Review

“…The sources of demonstrations can be kinesthetic teaching, teleoperation (Zahlner S. et al, (n.d.); Handa et al, 2019 ; Li T. et al, 2019 ; Li et al, 2020 ), raw video, and so on. The problem of learning from demonstration has been studied a lot in recent years and a comprehensive survey can be seen in Ramírez et al ( 2021 ). Ramírez et al ( 2021 ) divided the use of the demonstrations into two types of knowledge: prior knowledge and online knowledge.…”

“…The problem of learning from demonstration has been studied a lot in recent years and a comprehensive survey can be seen in Ramírez et al ( 2021 ). Ramírez et al ( 2021 ) divided the use of the demonstrations into two types of knowledge: prior knowledge and online knowledge. In the case of the former, the demonstration data were stored before the RL process and acted as source of knowledge such as being added to the reward function for bringing the policy closer to the demonstration.…”

Human-behavior learning: A new complementary learning perspective for optimal decision making controllers

“…Imitation learning [77] provides a wide number of techniques that enables the transference of knowledge from an expert to an apprentice. These techniques are very common in robotic applications such as learning by demonstrations [55,78] which uses several techniques such as Gaussian processes, dynamic timewarping [1], Lloyd's algorithm [79,80], among others.…”

“…3) for experience transference [6,81]. This complementary property can be achieved by means of inverse optimal control (IOC)( [82]) or inverse reinforcement learning (IRL) [83,77] which are discussed below.…”