Human-Centered Reinforcement Learning: A Survey

Li, Guangliang; Gómez, Randy; Nakamura, Keisuke; He, Bo

doi:10.1109/thms.2019.2912447

Cited by 94 publications

(40 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Integrating evaluative feedback from humans into RL is sometimes called human-centered RL. A survey on humancentered RL topic is provided by [37]. One important model to mention is TAMER [38], [39].…”

Section: Related Workmentioning

confidence: 99%

Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Bied

Chétouani

2020

2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)

View full text Add to dashboard Cite

An important aspect of Human-Robotcooperation is that the robot is capable of clearly communicating its intentions to its human collaborator. This communication of intentions often requires the generation of legible motion trajectories. The concept of legible motion is usually not studied together with machine learning. Studying these fields together is an important step towards better Human-Robot cooperation. In this paper, we investigate interactive robot learning approaches with the aim of developing models that are able to generate legible motions by taking observer feedback into account. We explore how to integrate the observer feedback into a Reinforcement Learning (RL) framework. We do this by proposing three different observer algorithms as observer strategies in an interactive RL scheme and compare with one non-interactive RL algorithm as baseline. For the observer strategies we vary the method how the observer estimates how likely the agent is going for the target goal. We evaluate our approach on five environments and calculate the legibility of the learned trajectories. The results show that the legibility of the learned trajectories is significantly higher while integrating the feedback from the observer compared with a standard Q-Learning algorithm not using the observer feedback.

show abstract

Section: Related Workmentioning

confidence: 99%

Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Bied

Chétouani

2020

2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)

View full text Add to dashboard Cite

show abstract

“…In the first one, called reward-shaping, the trainer modifies or accepts the reward given by the environment in order to bias the agent's learning [19,20]. In the second one, called policy-shaping, the trainer may suggest a different action to perform, by replacing the one proposes by the policy [21,22]. A simple policy-shaping method involves forcing the agent to take certain actions that are recommended by the trainer [23,24].…”

Section: Deep Reinforcement Learning and Interactive Feedbackmentioning

confidence: 99%

Deep Reinforcement Learning with Interactive Feedback in a Human–Robot Environment

et al. 2020

View full text Add to dashboard Cite

Robots are extending their presence in domestic environments every day, it being more common to see them carrying out tasks in home scenarios. In the future, robots are expected to increasingly perform more complex tasks and, therefore, be able to acquire experience from different sources as quickly as possible. A plausible approach to address this issue is interactive feedback, where a trainer advises a learner on which actions should be taken from specific states to speed up the learning process. Moreover, deep reinforcement learning has been recently widely used in robotics to learn the environment and acquire new skills autonomously. However, an open issue when using deep reinforcement learning is the excessive time needed to learn a task from raw input images. In this work, we propose a deep reinforcement learning approach with interactive feedback to learn a domestic task in a Human–Robot scenario. We compare three different learning methods using a simulated robotic arm for the task of organizing different objects; the proposed methods are (i) deep reinforcement learning (DeepRL); (ii) interactive deep reinforcement learning using a previously trained artificial agent as an advisor (agent–IDeepRL); and (iii) interactive deep reinforcement learning using a human advisor (human–IDeepRL). We demonstrate that interactive approaches provide advantages for the learning process. The obtained results show that a learner agent, using either agent–IDeepRL or human–IDeepRL, completes the given task earlier and has fewer mistakes compared to the autonomous DeepRL approach.

show abstract

“…An inefficient reward function means that the controller needs a lot of learning samples and time to test and explore, which seriously limits the application of the traditional reinforcement learning method to the actual robot system. Therefore, on the basis of traditional reinforcement learning, researchers put forward interactive reinforcement learning [7], [14], [16], [17], [19], [20], [28]- [30]. In interactive reinforcement learning, an agent learns in an MDP without reward function.…”

Section: A Deep Q-networkmentioning

confidence: 99%

“…In order to speed up the robot learning, researchers propose interactive reinforcement learning (IRL) [7] based on reward shaping [8] in traditional reinforcement learning methods. Interactive reinforcement learning allows designers and even non-technical personnel to train robots by evaluating their behavior.…”

Section: Introductionmentioning

confidence: 99%

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

et al. 2020

Self Cite

View full text Add to dashboard Cite

Autonomous underwater vehicle (AUV) plays an increasingly important role in ocean exploration. Existing AUVs are usually not fully autonomous and generally limited to pre-planning or preprogramming tasks. Reinforcement learning (RL) and deep reinforcement learning have been introduced into the AUV design and research to improve its autonomy. However, these methods are still difficult to apply directly to the actual AUV system because of the sparse rewards and low learning efficiency. In this paper, we proposed a deep interactive reinforcement learning method for path following of AUV by combining the advantages of deep reinforcement learning and interactive RL. In addition, since the human trainer cannot provide human rewards for AUV when it is running in the ocean and AUV needs to adapt to a changing environment, we further propose a deep reinforcement learning method that learns from both human rewards and environmental rewards at the same time. We test our methods in two path following tasks-straight line and sinusoids curve following of AUV by simulating in the Gazebo platform. Our experimental results show that with our proposed deep interactive RL method, AUV can converge faster than a DQN learner from only environmental reward. Moreover, AUV learning with our deep RL from both human and environmental rewards can also achieve a similar or even better performance than that with the deep interactive RL method and can adapt to the actual environment by further learning from environmental rewards. INDEX TERMS Autonomous Underwater Vehicle, Interactive Reinforcement Learning, Deep Q Network, Path FollowingGUANGLIANG LI (M'14) received the Bachelor's degree in automation and M.Sc. degree in control theory and control engineering from the

show abstract

Human-Centered Reinforcement Learning: A Survey

Cited by 94 publications

References 46 publications

Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Deep Reinforcement Learning with Interactive Feedback in a Human–Robot Environment

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

Contact Info

Product

Resources

About