Measuring and Characterizing Generalization in Deep Reinforcement Learning

Witty, Sam; Lee, Jun Ki; Tosch, Emma; Atrey, Akanksha; Littman, Michael L.; Jensen, David G.

doi:10.48550/arxiv.1812.02868

Cited by 9 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This means that any state we visit is actually reachable, at least in the training environment. The closest method to ours appears in the work of (Witty et al, 2018), in which the authors try to characterize generalization by exploring different starting configurations, either by directly changing the start state or by using states visited by another agent. In contrast to our work, their insights are applied to characterizing generalization, and not as explanations for humans to understand agent behavior.…”

Section: Counterfactual Style Explanationsmentioning

confidence: 99%

Explaining Reinforcement Learning Policies through Counterfactual Trajectories

Frost¹,

Watkins²,

Weiner³

et al. 2022

Preprint

View full text Add to dashboard Cite

In order for humans to confidently decide where to employ RL agents for real-world tasks, a human developer must validate that the agent will perform well at test-time. Some policy interpretability methods facilitate this by capturing the policy's decision making in a set of agent rollouts. However, even the most informative trajectories of training time behavior may give little insight into the agent's behavior out of distribution. In contrast, our method conveys how the agent performs under distribution shifts by showing the agent's behavior across a wider trajectory distribution. We generate these trajectories by guiding the agent to more diverse unseen states and showing the agent's behavior there. In a user study, we demonstrate that our method enables users to score better than baseline methods on one of two agent validation tasks.

show abstract

Section: Counterfactual Style Explanationsmentioning

confidence: 99%

Explaining Reinforcement Learning Policies through Counterfactual Trajectories

Frost¹,

Watkins²,

Weiner³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This behavior was magnified the more enemies were removed. This implied the agent had not learned a robust strategy for the game, but was simply using the enemies as a "signal" as to how to act; a more detailed analysis is available in Witty et al (2018).…”

Section: Approachmentioning

confidence: 99%

“…In control networks, where the output could govern the actions of highly impactful systems (e.g., autonomous vehicles), the lack of explainability is worrisome. Even more problematic, deep RL based systems have been shown to be highly dependent on the exact structure of the input, producing wildly different output with only small perturbations to the input Witty et al (2018). Users employing these system can be left with agents producing erratic behavior with little indication as to why.…”

Section: Introductionmentioning

confidence: 99%

Explainable Artificial Intelligence (XAI) for Increasing User Trust in Deep Reinforcement Learning Driven Autonomous Systems

Druce¹,

Harradon²,

Tittle³

2021

Preprint

View full text Add to dashboard Cite

We consider the problem of providing users of deep Reinforcement Learning (RL) based systems with a better understanding of when their output can be trusted. We offer an explainable artificial intelligence (XAI) framework that provides a three-fold explanation: a graphical depiction of the systems generalization and performance in the current game state, how well the agent would play in semantically similar environments, and a narrative explanation of what the graphical information implies. We created a user-interface for our XAI framework and evaluated its efficacy via a human-user experiment. The results demonstrate a statistically significant increase in user trust and acceptance of the AI system with explanation, versus the AI system without explanation.

show abstract

“…Model-free RL, like model-based RL, has also suffered from both the "train=test" paradigm and a lack of standardization around how to measure generalization. In response, recent papers have discussed what generalization in RL means and how to measure it [7,8,36,49,71], and others have proposed new environments such as Procgen [9] and Meta-World [74] as benchmarks focusing on measuring generalization. While popular in the model-free community [e.g.…”

Section: Introductionmentioning

confidence: 99%

Procedural Generalization by Planning with Self-Supervised World Models

Anand¹,

Walker²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

One of the key promises of model-based reinforcement learning is the ability to generalize using an internal model of the world to make predictions in novel environments and tasks. However, the generalization ability of model-based agents is not well understood because existing work has focused on model-free agents when benchmarking generalization. Here, we explicitly measure the generalization ability of model-based agents in comparison to their model-free counterparts. We focus our analysis on MuZero [60], a powerful model-based agent, and evaluate its performance on both procedural and task generalization. We identify three factors of procedural generalization-planning, self-supervised representation learning, and procedural data diversity-and show that by combining these techniques, we achieve state-of-the art generalization performance and data efficiency on Procgen [9]. However, we find that these factors do not always provide the same benefits for the task generalization benchmarks in Meta-World [74], indicating that transfer remains a challenge and may require different approaches than procedural generalization. Overall, we suggest that building generalizable agents requires moving beyond the single-task, model-free paradigm and towards self-supervised model-based agents that are trained in rich, procedural, multi-task environments.

show abstract

Measuring and Characterizing Generalization in Deep Reinforcement Learning

Cited by 9 publications

References 0 publications

Explaining Reinforcement Learning Policies through Counterfactual Trajectories

Explaining Reinforcement Learning Policies through Counterfactual Trajectories

Explainable Artificial Intelligence (XAI) for Increasing User Trust in Deep Reinforcement Learning Driven Autonomous Systems

Procedural Generalization by Planning with Self-Supervised World Models

Contact Info

Product

Resources

About