Strategic Tasks for Explainable Reinforcement Learning

Pocius, Rey; Neal, Lawrence; Fern, Alan

doi:10.1609/aaai.v33i01.330110007

Cited by 12 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most common approach to providing these explanations is to develop a model of the agent's behaviour using a separate observer that learns the agent's behaviour. There have been several generic explanation facilities that can perform this task, such as Pocius et al [177], which extends Local Interpretable Model-Agnostic Explanations (LIME) [183] and can provide contrastive explanations of any type of agents' behaviour -not solely an RL agent. These generic explanation facilities can predict behaviour, but do not explain the agent's internal reasoning for its behaviour.…”

Section: Results Of Xrl-behaviourmentioning

confidence: 99%

Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey

Dazeley¹,

Vamplew²,

Cruz³

2021

Preprint

View full text Add to dashboard Cite

Broad Explainable Artificial Intelligence (Broad-XAI) moves away from interpreting individual decisions based on a single datum and aims to provide integrated explanations from multiple machine learning algorithms into a coherent explanation of an agent's behaviour that is aligned to the communication needs of the explainee. Reinforcement Learning (RL) methods, we propose, provide a potential backbone for the cognitive model required for the development of Broad-XAI. RL represents a suite of approaches that have had increasing success in solving a range of sequential decision-making problems. However, these algorithms all operate as black-box problem solvers, where they obfuscate their decision-making policy through a complex array of values and functions. EXplainable RL (XRL) is relatively recent field of research that aims to develop techniques to extract concepts from the agent's: perception of the environment; intrinsic/extrinsic motivations/beliefs; Q-values, goals and objectives. This paper aims to introduce a conceptual framework, called the Causal XRL Framework (CXF), that unifies the current XRL research and uses RL as a backbone to the development of Broad-XAI. Additionally, we recognise that RL methods have the ability to incorporate a range of technologies to allow agents to adapt to their environment. CXF is designed for the incorporation of many standard RL extensions and integrated with external ontologies and communication facilities so that the agent can answer questions that explain outcomes and justify its decisions. This paper aims to: establish XRL as a distinct branch of Explainable Artificial Intelligence (XAI); introduce a conceptual framework for XRL; review existing approaches explaining agent behaviour; and identify opportunities for future research. Finally, this paper will discuss how additional information can be extracted and ultimately integrated into models of communication, facilitating the development of Broad-XAI.

show abstract

Section: Results Of Xrl-behaviourmentioning

confidence: 99%

Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey

Dazeley¹,

Vamplew²,

Cruz³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The reward decomposition approach has been proposed as the basis for a set of decision-making tasks in [4], in environments that naturally provide multiple reward signals: The goal is to enhance explainability by providing high-level abstractions for sequential tasks. [23] aim to enable an autonomous agent to reason over and answer questions about its underlying control logic 𝐿, independent of its internal representation.…”

Section: Distal Explanations For Explainable Reinforcement Learning P...mentioning

confidence: 99%

Explainable Deep Reinforcement Learning: State of the Art and Challenges

Vouros

2022

ACM Comput. Surv.

View full text Add to dashboard Cite

Interpretability, explainability and transparency are key issues to introducing Artificial Intelligence methods in many critical domains: This is important due to ethical concerns and trust issues strongly connected to reliability, robustness, auditability and fairness, and has important consequences towards keeping the human in the loop in high levels of automation, especially in critical cases for decision making, where both (human and the machine) play important roles. While the research community has given much attention to explainability of closed (or black) prediction boxes, there are tremendous needs for explainability of closed-box methods that support agents to act autonomously in the real world. Reinforcement learning methods, and especially their deep versions, are such closed-box methods. In this article we aim to provide a review of state of the art methods for explainable deep reinforcement learning methods, taking also into account the needs of human operators - i.e., of those that take the actual and critical decisions in solving real-world problems. We provide a formal specification of the deep reinforcement learning explainability problems, and we identify the necessary components of a general explainable reinforcement learning framework. Based on these, we provide a comprehensive review of state of the art methods, categorizing them in classes according to the paradigm they follow, the interpretable models they use, and the surface representation of explanations provided. The article concludes identifying open questions and important challenges.

show abstract

“…Moreover, Wang et al [47] proposed an explainable recommendation system using an RL framework. Pocius et al [31] utilized saliency maps as a way to explain agent decisions in a partially-observable game scenario. Thus, they focus on providing visual explanation with Deep RL.…”

Section: Environmentmentioning

confidence: 99%

“…As aforementioned, although there is increasing literature in different XAI subfields, such as explainable planners, interpretable RL, or explainable agency, just a few works are addressing the XRL challenge in robotic scenarios. In some of those works, although they are in a certain way focused on XRL, they have different aims than ours, e.g., to explain the learning process using saliency maps from a computational vision perspective, especially when using deep reinforcement learning as in [31]. In this paper, we focus on explaining goal-oriented decisions to provide an understanding to the user of what motivates the robot's specific actions from different states, taking into account the problem domain.…”

Section: Outcome-focused Explanationsmentioning

confidence: 99%

“…To get robots into our daily-life environments, an explainable robotic system should provide clear explanations specially focused on non-expert end-users in order to understand the robot's decisions. Often, such explanations, given by robotic systems, have been focused on interpreting the agent's decision based on its perception of the environment [31,21]. Whereas, little work has provided explanations of how the action selected is expected to help it to achieve its goal.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario

Cruz,

Dazeley,

Vamplew

et al. 2020

Preprint

View full text Add to dashboard Cite

Robotic systems are more present in our society everyday. In human-robot interaction scenarios, it is crucial that end-users develop trust in their robotic team-partners, in order to collaboratively complete a task. To increase trust, users demand more understanding about the decisions by the robot in particular situations. Recently, explainable robotic systems have emerged as an alternative focused not only on completing a task satisfactorily, but also in justifying, in a human-like manner, the reasons that lead to making a decision. In reinforcement learning scenarios, a great effort has been focused on providing explanations from the visual input modality, particularly using deep learning-based approaches. In this work, we focus on the decision-making process of a reinforcement learning agent performing a navigation task in a robotic scenario. As a way to explain the robot's behavior, we use the probability of success computed by three different proposed approaches: memory-based, learning-based, and phenomenological-based. The difference between these approaches is the additional memory required to compute or estimate the probability of success as well as the kind of reinforcement learning representation where they could be used. In this regard, we use the memory-based approach as a baseline since it is obtained directly from the agent's observations. When comparing the learning-based and the phenomenological-based approaches to this baseline, both are found to be suitable alternatives to compute the probability of success, obtaining high levels of similarity when compared using both the Pearson's correlation and the mean squared error.

show abstract

Strategic Tasks for Explainable Reinforcement Learning

Cited by 12 publications

References 3 publications

Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey

Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey

Explainable Deep Reinforcement Learning: State of the Art and Challenges

Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario

Contact Info

Product

Resources

About