Invariant Causal Prediction for Block MDPs

Zhang, Amy; Lyle, Clare; Sodhani, Shagun; Filos, Angelos; Kwiatkowska, Marta; Pineau, Joëlle; Gal, Yarin; Precup, Doina

doi:10.48550/arxiv.2003.06016

Cited by 9 publications

(17 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, it is reasonable that multitask learning (Caruana, 1997) has been used successfully across all applications of machine learning, because multitask data provide multiple views of their shared features (Nature Variables), making inferring them more accurate as suggested in MUA. Another example is that in reinforcement learning, one widely leveraged multiview data to discover the invariant part of states (Lu et al, 2018;Zhang et al, 2020).…”

Section: Implications Of a Unifying Viewmentioning

confidence: 99%

Nonlinear Invariant Risk Minimization: A Causal Approach

Lu¹,

Wu²,

Hernández-Lobato³

et al. 2021

Preprint

View full text Add to dashboard Cite

Due to spurious correlations, machine learning systems often fail to generalize to environments whose distributions differ from the ones used at training time. Prior work addressing this, either explicitly or implicitly, attempted to find a data representation that has an invariant causal relationship with the target. This is done by leveraging a diverse set of training environments to reduce the effect of spurious features and build an invariant predictor. However, these methods have generalization guarantees only when both data representation and classifiers come from a linear model class. We propose Invariant Causal Representation Learning (ICRL), a learning paradigm that enables out-of-distribution (OOD) generalization in the nonlinear setting (i.e., nonlinear representations and nonlinear classifiers). It builds upon a practical and general assumption: the prior over the data representation factorizes when conditioning on the target and the environment. Based on this, we show identifiability of the data representation up to very simple transformations. We also prove that all direct causes of the target can be fully discovered, which further enables us to obtain generalization guarantees in the nonlinear setting. Extensive experiments on both synthetic and real-world datasets show that our approach significantly outperforms a variety of baseline methods. Finally, in the concluding discussion, we further explore the aforementioned assumption and propose a general view, called the Agnostic Hypothesis: there exist a set of hidden causal factors affecting both inputs and outcomes. The Agnostic Hypothesis can provide a unifying view of machine learning, be it supervised, unsupervised, or reinforcement learning, in terms of representation learning. More importantly, it can inspire a new direction to explore the general theory for identifying hidden causal factors, which is key to enabling the OOD generalization guarantees in machine learning.

show abstract

Section: Implications Of a Unifying Viewmentioning

confidence: 99%

Nonlinear Invariant Risk Minimization: A Causal Approach

Lu¹,

Wu²,

Hernández-Lobato³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It not only aims to address the challenging OOD prediction problem, but also is a pioneer work that guides causal machine learning research towards the development of inductive bias imposing causal constraints. The effectiveness of IRM and its variants have been demonstrated across various areas including computer vision [1], natural language processing [5], CTR prediction [28], reinforcement learning [29] and financial forecasting [14]. [2] propose the original formulation of IRM as a two-stage optimization problem:…”

Section: Preliminaries and Irmmentioning

confidence: 99%

Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix

Guo¹,

Zhang²,

Líu³

et al. 2021

Preprint

View full text Add to dashboard Cite

This work considers the out-of-distribution (OOD) prediction problem where (1) the training data are from multiple domains and (2) the test domain is unseen in the training. DNNs fail in OOD prediction because they are prone to pick up spurious correlations. Recently, Invariant Risk Minimization (IRM) is proposed to address this issue. Its effectiveness has been demonstrated in the colored MNIST experiment. Nevertheless, we find that the performance of IRM can be dramatically degraded under strong Λ spuriousness -when the spurious correlation between the spurious features and the class label is strong due to the strong causal influence of their common cause, the domain label, on both of them (see Fig. 1). In this work, we try to answer the questions: why does IRM fail in the aforementioned setting? Why does IRM work for the original colored MNIST dataset? Then, we propose a simple and effective approach to fix the problem of IRM. We combine IRM with conditional distribution matching to avoid a specific type of spurious correlation under strong Λ spuriousness. Empirically, we design a series of semi synthetic datasetsthe colored MNIST plus, which exposes the problems of IRM and demonstrates the efficacy of the proposed method.

show abstract

“…Finally, the IRM method has recently been applied to RL problems. In [24], the authors attempt to learn a causal Markov decision process (MDP) that is bisimilar to the full MDP present during training. This formulation requires learning a model for both the causal and full dynamics of the system, a mapping between the two, and a causal model of the rewards.…”

Section: Related Workmentioning

confidence: 99%

Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

Sonar,

Pacelli,

Majumdar

2020

Preprint

View full text Add to dashboard Cite

A fundamental challenge in reinforcement learning is to learn policies that generalize beyond the operating domain experienced during training. In this paper, we approach this challenge through the following invariance principle: an agent must find a representation such that there exists an action-predictor built on top of this representation that is simultaneously optimal across all training domains. Intuitively, the resulting invariant policy enhances generalization by finding causes of successful actions. We propose a novel learning algorithm, Invariant Policy Optimization (IPO), that explicitly enforces this principle and learns an invariant policy during training. We compare our approach with standard policy gradient methods and demonstrate significant improvements in generalization performance on unseen domains for Linear Quadratic Regulator (LQR) problems and our own benchmark in the MiniGrid Gym environment.

show abstract

Invariant Causal Prediction for Block MDPs

Cited by 9 publications

References 10 publications

Nonlinear Invariant Risk Minimization: A Causal Approach

Nonlinear Invariant Risk Minimization: A Causal Approach

Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix

Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

Contact Info

Product

Resources

About