Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Abstract

An important aspect of Human-Robotcooperation is that the robot is capable of clearly communicating its intentions to its human collaborator. This communication of intentions often requires the generation of legible motion trajectories. The concept of legible motion is usually not studied together with machine learning. Studying these fields together is an important step towards better Human-Robot cooperation. In this paper, we investigate interactive robot learning approaches with the aim of developing models that are able to generate legible motions by taking observer feedback into account. We explore how to integrate the observer feedback into a Reinforcement Learning (RL) framework. We do this by proposing three different observer algorithms as observer strategies in an interactive RL scheme and compare with one non-interactive RL algorithm as baseline. For the observer strategies we vary the method how the observer estimates how likely the agent is going for the target goal. We evaluate our approach on five environments and calculate the legibility of the learned trajectories. The results show that the legibility of the learned trajectories is significantly higher while integrating the feedback from the observer compared with a standard Q-Learning algorithm not using the observer feedback.

Policy Regularization for Legible Behavior

“…Furthermore, we noticed that our results are obtained with a simple observer model having uniformly distributed probabilities, and that considers the same policies that are available to the agent. Because our results are qualitative similar to those reported in earlier works on legibility [11,26] ie. legible trajectories are skewed to avoid other goal locations, it is suggested that a similarly uniformly initialized observer model was implicitly utilized in those papers as well.…”

“…To the best of our knowledge very little work exists in RL relating to interpretable behavior as we just described. Both [26,27] propose methods relying on a transposition of the original formulation of legibility. The methods result applicable only for goal-driven policies, thus excluding all other types of policies available in various RL frameworks.…”

Policy regularization for legible behavior

“…Furthermore, we noticed that our results are obtained with a simple observer model that considers the same policies that are available to the agent but with a uniform probability. Because our results are qualitative similar to those reported in earlier works on legibility [11,27,28] i.e., legible trajectories are skewed to avoid other goal locations, is suggested that a similarly uniformly initialized observer model was implicitly utilized in those papers as well. However, we regularize the agent by working on policies rather than reward distributions, and for this reason the proposed method has three major advantages: firstly it easily generalizes over different shapes of reward regions and not only only on goal states that are a particular type of reward region.…”

“…Notice how this type of reward regions simulates goal locations, and thus allows to qualitatively confront the here obtained legible behavior with those for goal-driven policies from literature [27,28]. The behaviors are quite similar, with trajectories that are arced to disambiguate the goals.…”

“…To the best of our knowledge very little work exists in RL relating to interpretable behavior as we just described. Both [27,28] propose methods relying on a transposition of the original formulation of legibility. The methods result applicable only for goal-driven policies, thus excluding all other types of policies available in various RL frameworks.…”