Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

Sonar, Anoopkumar; Pacelli, Vincent; Majumdar, Anirudha

doi:10.48550/arxiv.2006.01096

Cited by 5 publications

(9 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several approaches use multiple contexts to learn an invariant representation, which is then assumed to generalise well to testing contexts. Sonar et al [140,IPO] apply ideas from Invariant Risk Minimization [141] to policy optimisation, learning a representation which enables jointly optimal action prediction across all domains, and show improved performance over PPO on several visual and dynamics variation environments. Bertrán et al [142,IAPE] introduce the Instance MDP, an alternative formalism for the generalisation problem, and then motivate theoretically an approach to learn a collection of policies on subsets of the training domains, such that the aggregate policy is invariant to any individual context-specific features which would not generalise.…”

Section: Much Work Draws On Causal Inference To Learn Invariant Featu...mentioning

confidence: 99%

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Kirk¹,

Zhang²,

Grefenstette³

et al. 2021

Preprint

View full text Add to dashboard Cite

The study of generalisation in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We provide a unifying formalism and terminology for discussing different generalisation problems, building upon previous works. We go on to categorise existing benchmarks for generalisation, as well as current methods for tackling the generalisation problem. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in generalisation, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for generalisation, and we recommend building benchmarks in underexplored problem settings such as offline RL generalisation and reward-function variation.

show abstract

Section: Much Work Draws On Causal Inference To Learn Invariant Featu...mentioning

confidence: 99%

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Kirk¹,

Zhang²,

Grefenstette³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Due to the new concept and easy implementation, IRM has gain notable visibility recently. There are some further theoretical analyses on the success [132] and failure cases of IRM [133], and IRM has been extended to other tasks including text classification [134] and reinforcement learning [135]. The idea to pursue the invariance of optimal representationlevel classifier is also extended.…”

Section: Domain-invariant Representation-based Dgmentioning

confidence: 99%

Generalizing to Unseen Domains: A Survey on Domain Generalization

Wang

Lan

Liu

et al. 2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

203

View full text Add to dashboard Cite

Domain generalization (DG), i.e., out-of-distribution generalization, has attracted increased interests in recent years. Domain generalization deals with a challenging setting where one or several different but related domain(s) are given, and the goal is to learn a model that can generalize to an unseen test domain. For years, great progress has been achieved. This paper presents the first review for recent advances in domain generalization. First, we provide a formal definition of domain generalization and discuss several related fields. Then, we categorize recent algorithms into three classes and present them in detail: data manipulation, representation learning, and learning strategy, each of which contains several popular algorithms. Third, we introduce the commonly used datasets and applications. Finally, we summarize existing literature and present some potential research topics for the future.

show abstract

“…We show how representations learned using PSM, when faced with semantically equivalent environments, can learn the main factors of variation and ignore spurious correlations that hinder generalization. We use LQR with distractors (Song et al, 2019;Sonar et al, 2020) to assess generalization in a feature-based RL setting with linear function approximation. The distractors are input features that are spuriously correlated with optimal actions and can be used for predicting these actions during training, but hurt generalization.…”

Section: Lqr With Spurious Correlationsmentioning

confidence: 99%

Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

Agarwal¹,

Machado²,

Castro³

et al. 2021

Preprint

View full text Add to dashboard Cite

Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments. To improve generalization, we incorporate the inherent sequential structure in reinforcement learning into the representation learning process. This approach is orthogonal to recent approaches, which rarely exploit this structure explicitly. Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states. PSM assigns high similarity to states for which the optimal policies in those states as well as in future states are similar. We also present a contrastive representation learning procedure to embed any state similarity metric, which we instantiate with PSM to obtain policy similarity embeddings (PSEs 1 ). We demonstrate that PSEs improve generalization on diverse benchmarks, including LQR with spurious correlations, a jumping task from pixels, and Distracting DM Control Suite. Source code would be made available at agarwl.github.io/pse.

show abstract

Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

Cited by 5 publications

References 30 publications

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Generalizing to Unseen Domains: A Survey on Domain Generalization

Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

Contact Info

Product

Resources

About