Multi-Channel Interactive Reinforcement Learning for Sequential Tasks

Abstract

The ability to learn new tasks by sequencing already known skills is an important requirement for future robots. Reinforcement learning is a powerful tool for this as it allows for a robot to learn and improve on how to combine skills for sequential tasks. However, in real robotic applications, the cost of sample collection and exploration prevent the application of reinforcement learning for a variety of tasks. To overcome these limitations, human input during reinforcement can be beneficial to speed up learning, guide the exploration and prevent the choice of disastrous actions. Nevertheless, there is a lack of experimental evaluations of multi-channel interactive reinforcement learning systems solving robotic tasks with input from inexperienced human users, in particular for cases where human input might be partially wrong. Therefore, in this paper, we present an approach that incorporates multiple human input channels for interactive reinforcement learning in a unified framework and evaluate it on two robotic tasks with 20 inexperienced human subjects. To enable the robot to also handle potentially incorrect human input we incorporate a novel concept for self-confidence, which allows the robot to question human input after an initial learning phase. The second robotic task is specifically designed to investigate if this self-confidence can enable the robot to achieve learning progress even if the human input is partially incorrect. Further, we evaluate how humans react to suggestions of the robot, once the robot notices human input might be wrong. Our experimental evaluations show that our approach can successfully incorporate human input to accelerate the learning process in both robotic tasks even if it is partially wrong. However, not all humans were willing to accept the robot's suggestions or its questioning of their input, particularly if they do not understand the learning process and the reasons behind the robot's suggestions. We believe that the findings from this experimental evaluation can be beneficial for the future design of algorithms and interfaces of interactive reinforcement learning systems used by inexperienced users.

Correct Me If I'm Wrong: Using Non-Experts to Repair Reinforcement Learning Policies

“…Such explicit rewards are used incorporate domain knowledge into RL [15], [16]. Alternatively, teacher intervention can serve as implicit negative rewards, indicating that actions are undesirable [17], [18].…”

Interactive Imitation Learning in Robotics: A Survey

“…Firstly, each type of human-robot interaction is more adequate to capture different stages of the learning process. Secondly, different types of information are used during the learning process, which can achieve a faster or more consistent learning rate (Koert et al, 2020).…”

“…This channel allows the robot to communicate the necessity of corrections or input, as in the case of robot-gated or active learning methods, in which the user is asked for feedback only in specific cases which improve the learning process (e.g., Sadigh et al, 2017;Cui and Niekum, 2018;Brown et al, 2018;Biyik and Sadigh, 2018;Biyik et al, 2020;Hoque et al, 2022;Franzese et al, 2021a). Additionally, Li et al (2016) and Koert et al (2020) show that providing the human with uncertainty and performance information about the robot's actions can improve the teacher's feedback quality, improving the learning process, i.e., the teacher concurrently learning how to teach the robot. Thus, understanding how and what information to communicate to the human can be a key to enabling non-experts to interactively teach robots.…”

Knowledge- and ambiguity-aware robot learning from corrective and evaluative feedback

“…Additionally, these interactive methods enable users to transfer their knowledge with other modalities of interaction, not only explicitly showing what the agent should do, but also guiding with evaluative feedback from the teacher, i.e., reinforcements (rewards or punishments), or also by comparing the performance of different agents/policies with learning from preferences or rankings. The use of evaluative feedback bridges the worlds of RL and IL [13][14][15][16].…”