Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario

Abstract

Robotic systems are more present in our society everyday. In human-robot interaction scenarios, it is crucial that end-users develop trust in their robotic team-partners, in order to collaboratively complete a task. To increase trust, users demand more understanding about the decisions by the robot in particular situations. Recently, explainable robotic systems have emerged as an alternative focused not only on completing a task satisfactorily, but also in justifying, in a human-like manner, the reasons that lead to making a decision. In reinforcement learning scenarios, a great effort has been focused on providing explanations from the visual input modality, particularly using deep learning-based approaches. In this work, we focus on the decision-making process of a reinforcement learning agent performing a navigation task in a robotic scenario. As a way to explain the robot's behavior, we use the probability of success computed by three different proposed approaches: memory-based, learning-based, and phenomenological-based. The difference between these approaches is the additional memory required to compute or estimate the probability of success as well as the kind of reinforcement learning representation where they could be used. In this regard, we use the memory-based approach as a baseline since it is obtained directly from the agent's observations. When comparing the learning-based and the phenomenological-based approaches to this baseline, both are found to be suitable alternatives to compute the probability of success, obtaining high levels of similarity when compared using both the Pearson's correlation and the mean squared error.

Incorporating Rivalry in Reinforcement Learning for a Competitive Game

“…To achieve this, each agent will compute the introspective confidence (ic a ) [38] of each action, which focuses on scaling the selected Q-value of an action towards the final goal using a logarithm transformation which computes the probability of success, in our case, of winning the game. The introspective confidence gives us a self-assessment of the agent's actions, based on its own game experience.…”

A Conceptual Framework for Externally-influenced Agents: An Assisted Reinforcement Learning Review

“…• Partially observable environments: In practice, many RL problems are partially observable [148]. For instance, partial observabilities may occur in non-stationary environments [39] or in presence of stochastic transitions [149]. If the external information source does not have observations to clearly infer the current state in the environment may lead to giving incorrect assistance to the learner agent.…”

Moody Learners -- Explaining Competitive Behaviour of Reinforcement Learning Agents

“…However, it is much more difficult to explain the agents' behavior in scenarios where the environment cannot be easily modeled [13], or the solutions are unknown a priori [14]. In this regard, recent applications derive human-level explanation based on the agent's own knowledge of the situation [15]. In particular, transforming the selected Q-values for each action into a confidence metric, using a re-shaping function based on the logarithmic transformation [15], improved the understanding of a robot trying to solve a grid-based navigation task.…”

“…In this regard, recent applications derive human-level explanation based on the agent's own knowledge of the situation [15]. In particular, transforming the selected Q-values for each action into a confidence metric, using a re-shaping function based on the logarithmic transformation [15], improved the understanding of a robot trying to solve a grid-based navigation task. The confidence metric measures how a specific action contributes to the robot reaching its goal however it does not carry any temporal correlation between the actions selected by the agent during the navigation.…”

“…To identify the impact of each selected action, we propose the Moody framework which uses the agents' own evaluation of the game to provide a temporal reference for the impact of selected actions and translate it into an pleasure/arousal representation [21] of the agent's mood. Our method builds on the introspective transformation from the Q-values [15], and introduces a Growing-When-Required (GWR) network to transform confidence readings into the pleasure/arousal scale, and to create strong inter-action correlation, which directly maps the action-selection-mood triplet in our competitive card game scenario and carries a temporal momentum that we use to explain the agents' performance while the game happens.…”