A Robust Approach for Continuous Interactive Reinforcement Learning

Millán-Arias, Cristian; Fernandes, Bruno; Cruz, Francisco; Dazeley, Richard; Fernandes, Stênio

doi:10.1145/3406499.3418769

Cited by 8 publications

(9 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human-sourced information has shown great potential due to its breadth, depth, and availability [ 9 ]. ARL agents that interact particularly with humans during operation are known as interactive agents, these agents have shown large improvements over unassisted agents [ 10 , 11 , 12 , 13 ].…”

Section: Introductionmentioning

confidence: 99%

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

et al. 2021

Self Cite

View full text Add to dashboard Cite

Interactive reinforcement learning methods utilise an external information source to evaluate decisions and accelerate learning. Previous work has shown that human advice could significantly improve learning agents’ performance. When evaluating reinforcement learning algorithms, it is common to repeat experiments as parameters are altered or to gain a sufficient sample size. In this regard, to require human interaction every time an experiment is restarted is undesirable, particularly when the expense in doing so can be considerable. Additionally, reusing the same people for the experiment introduces bias, as they will learn the behaviour of the agent and the dynamics of the environment. This paper presents a methodology for evaluating interactive reinforcement learning agents by employing simulated users. Simulated users allow human knowledge, bias, and interaction to be simulated. The use of simulated users allows the development and testing of reinforcement learning agents, and can provide indicative results of agent performance under defined human constraints. While simulated users are no replacement for actual humans, they do offer an affordable and fast alternative for evaluative assisted agents. We introduce a method for performing a preliminary evaluation utilising simulated users to show how performance changes depending on the type of user assisting the agent. Moreover, we describe how human interaction may be simulated, and present an experiment illustrating the applicability of simulating users in evaluating agent performance when assisted by different types of trainers. Experimental results show that the use of this methodology allows for greater insight into the performance of interactive reinforcement learning agents when advised by different users. The use of simulated users with varying characteristics allows for evaluation of the impact of those characteristics on the behaviour of the learning agent.

show abstract

Section: Introductionmentioning

confidence: 99%

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…For instance, advice may be used to construct or supplement the reward function, resulting in an improved evaluation of the agent's actions or increased the utility of the reward function requiring fewer experiences to learn a behaviour [30,31]. The advice can also be used to influence the agent's policy, either directly or through the action selection method, in order to reduce the search space [32,33].…”

Section: Reinforcement Learning and Interactive Human-sourced Advicementioning

confidence: 99%

Human engagement providing evaluative and informative advice for interactive reinforcement learning

Bignold

Cruz

Dazeley

et al. 2022

Neural Comput & Applic

Self Cite

View full text Add to dashboard Cite

Interactive reinforcement learning proposes the use of externally sourced information in order to speed up the learning process. When interacting with a learner agent, humans may provide either evaluative or informative advice. Prior research has focused on the effect of human-sourced advice by including real-time feedback on the interactive reinforcement learning process, specifically aiming to improve the learning speed of the agent, while minimising the time demands on the human. This work focuses on answering which of two approaches, evaluative or informative, is the preferred instructional approach for humans. Moreover, this work presents an experimental setup for a human trial designed to compare the methods people use to deliver advice in terms of human engagement. The results obtained show that users giving informative advice to the learner agents provide more accurate advice, are willing to assist the learner agent for a longer time, and provide more advice per episode. Additionally, self-evaluation from participants using the informative approach has indicated that the agent’s ability to follow the advice is higher, and therefore, they feel their own advice to be of higher accuracy when compared to people providing evaluative advice.

show abstract

“…where α ω is a learning rate of the disturber [56]. Algorithm 3 presents an episodic ADC with policy-gradients for the actor and the disturber.…”

Section: Robust Reinforcement Learningmentioning

confidence: 99%

“…In order to include advice during learning when the agent interacts with a dynamic environment, we additionaly combine the IRL and RRL approaches to propose Interactive Robust Reinforcement Learning (IRRL), an algorithm that involves advice for the agent to learn a task from an environment that has dynamic features [56]. Algorithm 4 shows the episodic ADC with advice in the context of IRRL.…”

Section: Interactive Robust Reinforcement Learning Approachmentioning

confidence: 99%

A Robust Approach for Continuous Interactive Actor-Critic Algorithms

et al. 2021

Self Cite

View full text Add to dashboard Cite

Reinforcement learning refers to a machine learning paradigm in which an agent interacts with the environment to learn how to perform a task. The characteristics of the environment may change over time or be affected by disturbances not controlled, avoiding the agent finding a proper policy. Some approaches attempt to address these problems, as interactive reinforcement learning, where an external entity helps the agent learn through advice. Other approaches, such as robust reinforcement learning, allow the agent to learn the task, acting in a disturbed environment. In this paper, we propose an approach that addresses interactive reinforcement learning problems in a dynamic environment, where advice provides information on the task and the dynamics of the environment. Thus, an agent learns a policy in a disturbed environment while receiving advice. We implement our approach in the dynamic version of the cart-pole balancing task and a simulated robotic arm dynamic environment to organize objects. Our results show that the proposed approach allows an agent to complete the task satisfactorily in a dynamic, continuous stateaction domain. Moreover, experimental results suggest agents trained with our approach are less sensitive to changes in the characteristics of the environment than interactive reinforcement learning agents.INDEX TERMS Continuous interactive reinforcement learning, interactive robust reinforcement learning, reinforcement learning, robust reinforcement learning.

show abstract

A Robust Approach for Continuous Interactive Reinforcement Learning

Cited by 8 publications

References 11 publications

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

Human engagement providing evaluative and informative advice for interactive reinforcement learning

A Robust Approach for Continuous Interactive Actor-Critic Algorithms

Contact Info

Product

Resources

About