Human engagement providing evaluative and informative advice for interactive reinforcement learning

Abstract

Interactive reinforcement learning proposes the use of externally sourced information in order to speed up the learning process. When interacting with a learner agent, humans may provide either evaluative or informative advice. Prior research has focused on the effect of human-sourced advice by including real-time feedback on the interactive reinforcement learning process, specifically aiming to improve the learning speed of the agent, while minimising the time demands on the human. This work focuses on answering which of two approaches, evaluative or informative, is the preferred instructional approach for humans. Moreover, this work presents an experimental setup for a human trial designed to compare the methods people use to deliver advice in terms of human engagement. The results obtained show that users giving informative advice to the learner agents provide more accurate advice, are willing to assist the learner agent for a longer time, and provide more advice per episode. Additionally, self-evaluation from participants using the informative approach has indicated that the agent’s ability to follow the advice is higher, and therefore, they feel their own advice to be of higher accuracy when compared to people providing evaluative advice.

An Advisor-Based Architecture for a Sample-Efficient Training of Autonomous Navigation Agents with Reinforcement Learning

“…Interactive reinforcement learning (IntRL) is an intriguing domain integrating a trainer to guide or evaluate the behavior of a learning agent [32,33]. The trainer's advice reinforces the agent's learning method and molds the exploration strategy, resulting in sample-efficient training by reducing the search spaces of states and actions [34]. Millan-Arias et al [35] developed an actor-critic RL agent that trained in a disturbed environment while receiving advice, allowing the agent to learn more robust policies.…”

Leveraging Expert Demonstration Features for Deep Reinforcement Learning in Floor Cleaning Robot Navigation

“…To solve the data querying problem, RL methods are still required. Methods without using IRL, but still infusing human supervision in RL setting, are proposed in [ 36 , 37 ]. Here, instead of obtaining the expert knowledge beforehand, a human-in-the-loop approach is used to query an expert for advice on action and reward.…”