A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms

Abstract

We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned while adapting to the change. It motivates using the speed of adaptation to a modified distribution as a meta-learning objective. We demonstrate how this can be used to determine the cause-effect relationship between two observed variables. The distributional changes do not need to correspond to standard interventions (clamping a variable), and the learner has no direct knowledge of these interventions. We show that causal structures can be parameterized via continuous variables and learned end-to-end. We then explore how these ideas could be used to also learn an encoder that would map low-level observed variables to unobserved causal variables leading to faster adaptation out-of-distribution, learning a representation space where one can satisfy the assumptions of independent mechanisms and of small and sparse changes in these mechanisms due to actions and non-stationarities.

Generalizing Graph Neural Networks on Out-Of-Distribution Graphs

“…hypothesis, i.e., training and testing graphs are independently sampled from the identical distribution (Liao et al, 2020). However, in reality, such a hypothesis is hardly satisfied due to the uncontrollable generation mechanism of real data, such as data selection biases, confounder factors or other peculiarities (Bengio et al, 2019;Engstrom et al, 2019;Su et al, 2019;Hendrycks & Dietterich, 2019). The testing distribution may incur uncontrolled and unknown shifts from the training distribution, called Out-Of-Distribution (OOD) shifts (Sun et al, 2019;Krueger et al, 2021), which makes most GNN models fail to make stable predictions.…”

Learning Neural Causal Models with Active Interventions

“…As a result, there has been a surge in interest in differentiable structure learning and the combination of deep learning and causal inference . Such methods define a structural causal model with smoothly differentiable parameters that are adjusted to fit observational data (Zheng et al, 2018;Yu et al, 2019;Zheng et al, 2020;Bengio et al, 2019;Lorch et al, 2021;Annadani et al, 2021), although some methods can accept interventional data, thereby significantly improving the identification of the underlying data-generating process Lippe et al, 2021). However, the improvement critically depends on the experiments and interventions available.…”

Efficient Neural Causal Discovery without Acyclicity Constraints

“…Uncovering and understanding causal mechanisms is an important problem not only in machine learning [3,31,39] but also in various scientific disciplines such as computational biology [10,28,37], epidemiology [36,46], and economics [15,31]. A common task of interest is causal structure learning [31,33] which aims at learning a directed acyclic graph (DAG) in which edges represent causal relations between variables.…”

“…Finding the right DAG is challenging as the solution space grows super-exponentially with the number of variables. A promising new direction are continuous-optimization methods [3,4,18,51,52,53,54] that are more computationally efficient than previous score-based and constraint-based methods [12,33] while leveraging the expressiveness of neural networks as function approximators. To restrict the search space to acyclic graphs, Zheng et al [52] first proposed to view the search as a constrained optimization problem using an augmented Lagrangian procedure to solve it.…”