Visual Rationalizations in Deep Reinforcement Learning for Atari Games

Weitkamp, Laurens; Pol, Elise van der; Akata, Zeynep

doi:10.1007/978-3-030-31978-6_12

Cited by 14 publications

(12 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used the Atari game Pacman for our experiments (see section 5 for the specific implementation). Atari games are a common benchmark for state of the art reinforcement learning algorithms [14,20,51,76] and to test explanation methods for those algorithms [5,28,36,44,77]. We chose Pacman since it is not as reaction-based as some other Atari games (e.g.…”

Section: Methodsmentioning

confidence: 99%

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

Huber

Weitz

André

et al. 2021

Artificial Intelligence

View full text Add to dashboard Cite

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as the environments in which they act have large state spaces, and their decision-making can be affected by delayed rewards, making it difficult to analyze their behavior. To address this problem, several approaches have been developed. Some approaches attempt to convey the global behavior of the agent, describing the actions it takes in different states. Other approaches devised local explanations which provide information regarding the agent's decision-making in a particular state. In this paper, we combine global and local explanation methods, and evaluate their joint and separate contributions, providing (to the best of our knowledge) the first user study of combined local and global explanations for RL agents. Specifically, we augment strategy summaries that extract important trajectories of states from simulations of the agent with saliency maps which show what information the agent attends to. Our results show that the choice of what states to include in the summary (global information) strongly affects people's understanding of agents: participants shown summaries that included important states significantly outperformed participants who were presented with agent behavior in a randomly set of chosen world-states. We find mixed results with respect to augmenting demonstrations with saliency maps (local information), as the addition of saliency maps did not significantly improve performance in most cases. However, we do find some evidence that saliency maps can help users better understand what information the agent relies on in its decision making, suggesting avenues for future work that can further improve explanations of RL agents.

show abstract

Section: Methodsmentioning

confidence: 99%

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

Huber

Weitz

André

et al. 2021

Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…Gradientbased methods compute saliency maps by estimating the input features' influence on the output using the gradient information (Simonyan, Vedaldi, and Zisserman 2013;Springenberg et al 2014;Mahendran and Vedaldi 2016;Zhang et al 2018a;Shrikumar, Greenside, and Kundaje 2017;Sundararajan, Taly, and Yan 2017;Selvaraju et al 2017;Chattopadhay et al 2018;Zhou et al 2016). These methods are for visualizing general DNNs but have been used to interpret deep RL agents (Joo and Kim 2019;Weitkamp, van der Pol, and Akata 2018;Shi et al 2020;Jaunet, Vuillemot, and Wolf 2019;Wang et al 2018). We did not use gradient-based saliency maps for our analysis because they lack physical meaning and could be difficult to interpret.…”

Section: Related Workmentioning

confidence: 99%

Leveraging Human Guidance for Deep Reinforcement Learning Tasks

Zhang

Torabi

Guan

et al. 2019

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Reinforcement learning agents can learn to solve sequential decision tasks by interacting with the environment. Human knowledge of how to solve these tasks can be incorporated using imitation learning, where the agent learns to imitate human demonstrated decisions. However, human guidance is not limited to the demonstrations. Other types of guidance could be more suitable for certain tasks and require less human effort. This survey provides a high-level overview of five recent learning frameworks that primarily rely on human guidance other than conventional, step-by-step action demonstrations. We review the motivation, assumption, and implementation of each framework. We then discuss possible future research directions.

show abstract

“…Zahavy et al [23] and Wang et al [20] for example used gradient-based saliency maps similar to [17] on traditional and Dueling DQN algorithms. Weitkamp et al [21] tested Grad-CAM on an Actor-Critic DRL algorithm. LRP has been used to visualize DRL in [10] but, to our knowledge, it has not been used to visualize the Dueling DQN architecture yet.…”

Section: Related Workmentioning

confidence: 99%

“…For a long time, DRL research only focused on optimizing the performance of DRL agents, but recent years saw an increasing interest in making the decision process of DRL agents more explainable [23,9,19,7,21]. One problem with explaining the actions of a DRL agent is that the inner workings of the underlying DNNs are incomprehensible to humans, making it difficult to identify the parts of the input on which the agent bases its decision.…”

Section: Introductionmentioning

confidence: 99%

Enhancing Explainability of Deep Reinforcement Learning Through Selective Layer-Wise Relevance Propagation

Huber

Schiller

André

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Modern deep reinforcement learning agents are capable of achieving superhuman performance in tasks like playing Atari games, solely based on visual input. However, due to their use of neural networks the trained models are lacking transparency which makes their inner workings incomprehensible for humans. A promising approach to gain insights into the opaque reasoning process of neural networks is the layer-wise relevance propagation (LRP) concept. This visualization technique creates saliency maps that highlight the areas in the input which were relevant for the agents' decision-making process. Since such saliency maps cover every possible cause for a prediction, they are often accentuating very diverse parts of the input. This makes the results difficult to understand for people without a machine-learning background. In this work, we introduce an adjustment to the LRP concept that utilizes only the most relevant neurons of each convolutional layer and thus generates more selective saliency maps. We test our approach with a dueling Deep Q-Network (DQN) agent which we trained on three different Atari games of varying complexity. Since the dueling DQN approach considerably alters the neural network architecture of the original DQN algorithm, it requires its own LRP variant which will be presented in this paper.

show abstract

Visual Rationalizations in Deep Reinforcement Learning for Atari Games

Cited by 14 publications

References 16 publications

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

Leveraging Human Guidance for Deep Reinforcement Learning Tasks

Enhancing Explainability of Deep Reinforcement Learning Through Selective Layer-Wise Relevance Propagation

Contact Info

Product

Resources

About