Deep Reinforcement Learning with Interactive Feedback in a Human–Robot Environment

Moreira, Ithan; Rivas, Javier; Cruz, Francisco; Dazeley, Richard; Ayala, Angel; Fernandes, Bruno

doi:10.3390/app10165574

Cited by 37 publications

(29 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Human-sourced information has shown great potential due to its breadth, depth, and availability [ 9 ]. ARL agents that interact particularly with humans during operation are known as interactive agents, these agents have shown large improvements over unassisted agents [ 10 , 11 , 12 , 13 ].…”

Section: Introductionmentioning

confidence: 99%

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

et al. 2021

Self Cite

View full text Add to dashboard Cite

Interactive reinforcement learning methods utilise an external information source to evaluate decisions and accelerate learning. Previous work has shown that human advice could significantly improve learning agents’ performance. When evaluating reinforcement learning algorithms, it is common to repeat experiments as parameters are altered or to gain a sufficient sample size. In this regard, to require human interaction every time an experiment is restarted is undesirable, particularly when the expense in doing so can be considerable. Additionally, reusing the same people for the experiment introduces bias, as they will learn the behaviour of the agent and the dynamics of the environment. This paper presents a methodology for evaluating interactive reinforcement learning agents by employing simulated users. Simulated users allow human knowledge, bias, and interaction to be simulated. The use of simulated users allows the development and testing of reinforcement learning agents, and can provide indicative results of agent performance under defined human constraints. While simulated users are no replacement for actual humans, they do offer an affordable and fast alternative for evaluative assisted agents. We introduce a method for performing a preliminary evaluation utilising simulated users to show how performance changes depending on the type of user assisting the agent. Moreover, we describe how human interaction may be simulated, and present an experiment illustrating the applicability of simulating users in evaluating agent performance when assisted by different types of trainers. Experimental results show that the use of this methodology allows for greater insight into the performance of interactive reinforcement learning agents when advised by different users. The use of simulated users with varying characteristics allows for evaluation of the impact of those characteristics on the behaviour of the learning agent.

show abstract

Section: Introductionmentioning

confidence: 99%

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…We implement a simulated robot arm for the task of organizing different objects [57]. The objective is to classify six geometric figures with different shapes and colors.…”

Section: B Simulated Robotic Armmentioning

confidence: 99%

“…until x t is terminal 17: end for and a simulated robotic arm for the task of organizing different objects [57].…”

mentioning

confidence: 99%

A Robust Approach for Continuous Interactive Actor-Critic Algorithms

et al. 2021

Self Cite

View full text Add to dashboard Cite

Reinforcement learning refers to a machine learning paradigm in which an agent interacts with the environment to learn how to perform a task. The characteristics of the environment may change over time or be affected by disturbances not controlled, avoiding the agent finding a proper policy. Some approaches attempt to address these problems, as interactive reinforcement learning, where an external entity helps the agent learn through advice. Other approaches, such as robust reinforcement learning, allow the agent to learn the task, acting in a disturbed environment. In this paper, we propose an approach that addresses interactive reinforcement learning problems in a dynamic environment, where advice provides information on the task and the dynamics of the environment. Thus, an agent learns a policy in a disturbed environment while receiving advice. We implement our approach in the dynamic version of the cart-pole balancing task and a simulated robotic arm dynamic environment to organize objects. Our results show that the proposed approach allows an agent to complete the task satisfactorily in a dynamic, continuous stateaction domain. Moreover, experimental results suggest agents trained with our approach are less sensitive to changes in the characteristics of the environment than interactive reinforcement learning agents.INDEX TERMS Continuous interactive reinforcement learning, interactive robust reinforcement learning, reinforcement learning, robust reinforcement learning.

show abstract

“…DRL has the advantages of not requiring environmental maps, strong learning capabilities, and high dynamic adaptability. Its deep network can successfully process high-dimensional information from sensors, and its reinforcement learning mechanism can perform continuous decision-making tasks in complex environments [ 16 , 17 , 18 ]. The successfully trained network model can directly generate optimal control commands for the lunar rover based on sensor information, omitting the complex environment reconstruction and sensing steps of traditional algorithms, and is very suitable for dynamic planning tasks such as lunar exploration with the unknown environment and limited on-board computing resources.…”

Section: Introductionmentioning

confidence: 99%

Learning-Based End-to-End Path Planning for Lunar Rovers with Safety Constraints

Wang

Zhang

2021

Sensors

View full text Add to dashboard Cite

Path planning is an essential technology for lunar rover to achieve safe and efficient autonomous exploration mission, this paper proposes a learning-based end-to-end path planning algorithm for lunar rovers with safety constraints. Firstly, a training environment integrating real lunar surface terrain data was built using the Gazebo simulation environment and a lunar rover simulator was created in it to simulate the real lunar surface environment and the lunar rover system. Then an end-to-end path planning algorithm based on deep reinforcement learning method is designed, including state space, action space, network structure, reward function considering slip behavior, and training method based on proximal policy optimization. In addition, to improve the generalization ability to different lunar surface topography and different scale environments, a variety of training scenarios were set up to train the network model using the idea of curriculum learning. The simulation results show that the proposed planning algorithm can successfully achieve the end-to-end path planning of the lunar rover, and the path generated by the proposed algorithm has a higher safety guarantee compared with the classical path planning algorithm.

show abstract

Deep Reinforcement Learning with Interactive Feedback in a Human–Robot Environment

Cited by 37 publications

References 28 publications

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

A Robust Approach for Continuous Interactive Actor-Critic Algorithms

Learning-Based End-to-End Path Planning for Lunar Rovers with Safety Constraints

Contact Info

Product

Resources

About