Exploring Computational User Models for Agent Policy Summarization

Abstract

AI agents are being developed to support high stakes decision-making processes from driving cars to prescribing drugs, making it increasingly important for human users to understand their behavior. Policy summarization methods aim to convey strengths and weaknesses of such agents by demonstrating their behavior in a subset of informative states. Some policy summarization methods extract a summary that optimizes the ability to reconstruct the agent's policy under the assumption that users will deploy inverse reinforcement learning. In this paper, we explore the use of different models for extracting summaries. We introduce an imitation learning-based approach to policy summarization; we demonstrate through computational simulations that a mismatch between the model used to extract a summary and the model used to reconstruct the policy results in worse reconstruction quality; and we demonstrate through a human-subject study that people use different models to reconstruct policies in different contexts, and that matching the summary extraction model to these can improve performance. Together, our results suggest that it is important to carefully consider user models in policy summarization.

Machine Teaching for Human Inverse Reinforcement Learning

“…Responses were far more likely to be coded as IRL in US2, where participants got to see five demonstrations as opposed to US1, where participants only got to see two or three demonstrations. This echoes the observation of Lage et al (2019) that people may be more inclined to use IL over IRL in less familiar situations, which may be moderated in future studies through more extensive pre-study practice and/or additional informative demonstrations that better familiarize the participant to the domains. Finally, out of 15 participants who provided more than one response, coders agreed that eight appeared to employ the same learning style throughout the user study (e.g.…”

Reinforcement Learning Interpretation Methods: A Survey

“…A summary is generated to describe the agent's interaction with the environment in different formats. Some summarization methods aim to describe the agent's policy in a limited set of states while others try to state the minimum possible information enough to reconstruct the agent's model by humans [74]. Amir et al [75] studied different methods in literature and proposed a conceptual framework for building policy summarization systems.…”

“…Instead of applying a method to extract the important states or state-action pairs, we assume that humans have some implicit models for the RL agent, and then we try to give them the minimum possible amount of information enough to make their model more representative to the actual agent model. In other words, we think of a summary as a medium that can be used by humans to reconstruct the agent's model [74], [80], [81]. This problem is the opposite of Inverse Reinforcement Learning (IRL) where we aim to make the RL imitate the user's model [81].…”

“…Moreover, Lage et al [74], [80] used the reconstruction accuracy to evaluate different summary extractions methods generated by simulation and then conducted experiments that involve humans to test their methods on the real world.…”

The Emerging Landscape of Explainable Automated Planning & Decision Making

“…A related work [16], looks at the use of soft decisions trees to create hierarchical representations of policies. A different approach is taken in [47] where users are presented partial plans that they can figure out completions of, based on their knowledge of the task. This is done by using various psychologically feasible computational models.…”