Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

Abstract

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as the environments in which they act have large state spaces, and their decision-making can be affected by delayed rewards, making it difficult to analyze their behavior. To address this problem, several approaches have been developed. Some approaches attempt to convey the global behavior of the agent, describing the actions it takes in different states. Other approaches devised local explanations which provide information regarding the agent's decision-making in a particular state. In this paper, we combine global and local explanation methods, and evaluate their joint and separate contributions, providing (to the best of our knowledge) the first user study of combined local and global explanations for RL agents. Specifically, we augment strategy summaries that extract important trajectories of states from simulations of the agent with saliency maps which show what information the agent attends to. Our results show that the choice of what states to include in the summary (global information) strongly affects people's understanding of agents: participants shown summaries that included important states significantly outperformed participants who were presented with agent behavior in a randomly set of chosen world-states. We find mixed results with respect to augmenting demonstrations with saliency maps (local information), as the addition of saliency maps did not significantly improve performance in most cases. However, we do find some evidence that saliency maps can help users better understand what information the agent relies on in its decision making, suggesting avenues for future work that can further improve explanations of RL agents.

Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents

“…The computational measurements done in this work present a first step to fully evaluate perturbation-based saliency maps for DRL. In the future, we will build upon the insights from this paper and conduct a human user-study, similar to the one we did in (Huber et al 2020), to evaluate how useful the saliency map approaches with good computational results are for actual end-users.…”

“…The evaluation metrics for XAI approaches can be separated into two broad categories: human user-studies and computational measurements (Mohseni, Zarei, and Ragan 2020). Examples of human user-studies of saliency maps for DRL agents are (Huber et al 2020) and (Anderson et al 2019), who evaluate LRP and Noise Sensitivity saliency maps respectively, with regards to mental models, trust and user satisfaction. To obtain more objective quantitative data it is important to additionally evaluate explanations through computational measurements.…”

Reinforcement Learning in Practice: Opportunities and Challenges

“…See Lipton (2018), Murdoch et al (2019), Rudin et al (2021) for discussions and surveys about interpretable machine learning. See some recent work on explainable RL, e.g., Amir et al (2019), Atrey et al (2020), Gottesman et al (2020), Hayes and Shah (2017), Huang et al (2019), Huber et al (2021), and Sequeira and Gervasio (2020.…”

Local Explanations for Reinforcement Learning

“…These approaches all intend to provide a global summary of the policy. Other summaries output trajectories deemed important according to importance measures [3,17] or through imitation learning [21], or train finite state representations to summarize a policy with an explainable model [7,8]. Visualization techniques combined with saliency have been used to either aggregate states and view the policy from a different perspective [49] or create a trajectory of saliency maps [14].…”

“…reduce dimension) as a function of actions [4] or formulas derived over factors of the state space [43] to output a policy summary, whereas we aggregate based on locality of the states determined by the expert policy dynamics and further identify strategic states based on these dynamics. Other summarization methods simply output simulated trajectories deemed important [3,17] as judged by whether or not the action taken at some state matters. We use the term policy dynamics to refer to state transitions and high probability paths.…”