Improving interactive reinforcement learning: What makes a good teacher?

Cruz, Francisco; Magg, Sven; Nagai, Yukie; Wermter, Stefan

doi:10.1080/09540091.2018.1443318

Cited by 43 publications

(40 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 1 shows the IntRL approach with an advisor observing the learning process and providing advice in selected episodes. While the human-in-the-loop approach to learning is one of the interactive agent’s greatest strength, the human can also often be the biggest obstacle [ 16 ]. Moreover, human trials are expensive, time-consuming, suffer from issues with repeatability, and acquisition of participants can be difficult.…”

Section: Introductionmentioning

confidence: 99%

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

et al. 2021

Self Cite

View full text Add to dashboard Cite

Interactive reinforcement learning methods utilise an external information source to evaluate decisions and accelerate learning. Previous work has shown that human advice could significantly improve learning agents’ performance. When evaluating reinforcement learning algorithms, it is common to repeat experiments as parameters are altered or to gain a sufficient sample size. In this regard, to require human interaction every time an experiment is restarted is undesirable, particularly when the expense in doing so can be considerable. Additionally, reusing the same people for the experiment introduces bias, as they will learn the behaviour of the agent and the dynamics of the environment. This paper presents a methodology for evaluating interactive reinforcement learning agents by employing simulated users. Simulated users allow human knowledge, bias, and interaction to be simulated. The use of simulated users allows the development and testing of reinforcement learning agents, and can provide indicative results of agent performance under defined human constraints. While simulated users are no replacement for actual humans, they do offer an affordable and fast alternative for evaluative assisted agents. We introduce a method for performing a preliminary evaluation utilising simulated users to show how performance changes depending on the type of user assisting the agent. Moreover, we describe how human interaction may be simulated, and present an experiment illustrating the applicability of simulating users in evaluating agent performance when assisted by different types of trainers. Experimental results show that the use of this methodology allows for greater insight into the performance of interactive reinforcement learning agents when advised by different users. The use of simulated users with varying characteristics allows for evaluation of the impact of those characteristics on the behaviour of the learning agent.

show abstract

Section: Introductionmentioning

confidence: 99%

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…A bad teacher can negatively influence the learning process and somehow limit the learner by teaching a strategy that is not necessarily optimal. To select a good teacher, it is necessary to take into account that an agent that obtains the best results for the task, in terms of accumulated reward, is not necessarily the best teacher [27]. Instead, a good teacher could be one with a small standard deviation over the visited states.…”

Section: Discussionmentioning

confidence: 99%

“…The quality of the given advice by the external trainer must also be considered to improve the learning. It has been shown that inconsistent advice may be very detrimental during the learning process, so that in case of low consistency of advice, autonomous learning may lead to better performance [27].…”

Section: Agent External Trainermentioning

confidence: 99%

Deep Reinforcement Learning with Interactive Feedback in a Human–Robot Environment

et al. 2020

Self Cite

View full text Add to dashboard Cite

Robots are extending their presence in domestic environments every day, it being more common to see them carrying out tasks in home scenarios. In the future, robots are expected to increasingly perform more complex tasks and, therefore, be able to acquire experience from different sources as quickly as possible. A plausible approach to address this issue is interactive feedback, where a trainer advises a learner on which actions should be taken from specific states to speed up the learning process. Moreover, deep reinforcement learning has been recently widely used in robotics to learn the environment and acquire new skills autonomously. However, an open issue when using deep reinforcement learning is the excessive time needed to learn a task from raw input images. In this work, we propose a deep reinforcement learning approach with interactive feedback to learn a domestic task in a Human–Robot scenario. We compare three different learning methods using a simulated robotic arm for the task of organizing different objects; the proposed methods are (i) deep reinforcement learning (DeepRL); (ii) interactive deep reinforcement learning using a previously trained artificial agent as an advisor (agent–IDeepRL); and (iii) interactive deep reinforcement learning using a human advisor (human–IDeepRL). We demonstrate that interactive approaches provide advantages for the learning process. The obtained results show that a learner agent, using either agent–IDeepRL or human–IDeepRL, completes the given task earlier and has fewer mistakes compared to the autonomous DeepRL approach.

show abstract

“…Torrey and Taylor ( 2013 ) report that different ways of advising, such as early, importance, mistake correcting, and predictive advising may become beneficial when teaching on a budget. Cruz et al ( 2018a ) studied which types of advisors are most beneficial during learning of the agent in a simulated domestic scenario.…”

Section: Related Workmentioning

confidence: 99%

Multi-Channel Interactive Reinforcement Learning for Sequential Tasks

et al. 2020

View full text Add to dashboard Cite

The ability to learn new tasks by sequencing already known skills is an important requirement for future robots. Reinforcement learning is a powerful tool for this as it allows for a robot to learn and improve on how to combine skills for sequential tasks. However, in real robotic applications, the cost of sample collection and exploration prevent the application of reinforcement learning for a variety of tasks. To overcome these limitations, human input during reinforcement can be beneficial to speed up learning, guide the exploration and prevent the choice of disastrous actions. Nevertheless, there is a lack of experimental evaluations of multi-channel interactive reinforcement learning systems solving robotic tasks with input from inexperienced human users, in particular for cases where human input might be partially wrong. Therefore, in this paper, we present an approach that incorporates multiple human input channels for interactive reinforcement learning in a unified framework and evaluate it on two robotic tasks with 20 inexperienced human subjects. To enable the robot to also handle potentially incorrect human input we incorporate a novel concept for self-confidence, which allows the robot to question human input after an initial learning phase. The second robotic task is specifically designed to investigate if this self-confidence can enable the robot to achieve learning progress even if the human input is partially incorrect. Further, we evaluate how humans react to suggestions of the robot, once the robot notices human input might be wrong. Our experimental evaluations show that our approach can successfully incorporate human input to accelerate the learning process in both robotic tasks even if it is partially wrong. However, not all humans were willing to accept the robot's suggestions or its questioning of their input, particularly if they do not understand the learning process and the reasons behind the robot's suggestions. We believe that the findings from this experimental evaluation can be beneficial for the future design of algorithms and interfaces of interactive reinforcement learning systems used by inexperienced users.

show abstract

Improving interactive reinforcement learning: What makes a good teacher?

Cited by 43 publications

References 25 publications

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

Deep Reinforcement Learning with Interactive Feedback in a Human–Robot Environment

Multi-Channel Interactive Reinforcement Learning for Sequential Tasks

Contact Info

Product

Resources

About