Weakly supervised causal representation learning

Brehmer, Johann; Haan, Pim de; Lippe, Phillip; Cohen, Taco

doi:10.48550/arxiv.2203.16437

Cited by 4 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ICLM learns by maximising an ELBO approximation to the maximum likelihood, and a noise decoder p(x | ) generates a sample from the inferred exogenous variables. Brehmer et al [100] reproduce interventional distributions X I ∼ p(x | do(a)) as demonstrated in Fig. 4.5.…”

Section: Weakly Supervised Causal Disentanglementmentioning

confidence: 66%

“…Brehmer et al [100] improve upon Ada-GVAE by removing the requirement that a practitioner has access to the number of intervened variables between sample pairs, thus propose to learn a generative model over a collection of tuples {(x i , x I )} n i=1 . In this setting, they propose two types of models: Explicit or Implicit Latent Causal Models.…”

Section: Weakly Supervised Causal Disentanglementmentioning

confidence: 99%

See 1 more Smart Citation

Causal Machine Learning: A Survey and Open Problems

Jean¹,

Lynch²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Causal Machine Learning (CausalML) is an umbrella term for machine learning methods that formalize the data-generation process as a structural causal model (SCM). This allows one to reason about the effects of changes to this process (i.e., interventions) and what would have happened in hindsight (i.e., counterfactuals). We categorize work in CausalML into five groups according to the problems they tackle: (1) causal supervised learning, (2) causal generative modeling, (3) causal explanations, (4) causal fairness, (5) causal reinforcement learning. For each category, we systematically compare its methods and point out open problems. Further, we review modality-specific applications in computer vision, natural language processing, and graph representation learning. Finally, we provide an overview of causal benchmarks and a critical discussion of the state of this nascent field, including recommendations for future work.

show abstract

Section: Weakly Supervised Causal Disentanglementmentioning

confidence: 66%

Section: Weakly Supervised Causal Disentanglementmentioning

confidence: 99%

Causal Machine Learning: A Survey and Open Problems

Jean¹,

Lynch²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Lippe et al exploit causal interventions on the latents to provide identification guarantees but require the knowledge of intervention targets and assume an invariant causal model describing the relations between any adjacent time frames. In concurrent work, Brehmer et al (2022) leverage data generated under causal interventions as a source of weak supervision and prove identification for structural causal models that are diffeomorphic transforms of exogenous noise. In addition to the above, there are a number of recent papers that explain the success of self-supervised contrastive learning through the lens of identification of representations.…”

Section: Related Workmentioning

confidence: 99%

Weakly Supervised Representation Learning with Sparse Perturbations

Ahuja¹,

Hartford²,

Bengio³

2022

Preprint

View full text Add to dashboard Cite

The theory of representation learning aims to build methods that provably invert the data generating process with minimal domain knowledge or any source of supervision. Most prior approaches require strong distributional assumptions on the latent variables and weak supervision (auxiliary information such as timestamps) to provide provable identification guarantees. In this work, we show that if one has weak supervision from observations generated by sparse perturbations of the latent variables-e.g. images in a reinforcement learning environment where actions move individual sprites-identification is achievable under unknown continuous latent distributions. We show that if the perturbations are applied only on mutually exclusive blocks of latents, we identify the latents up to those blocks. We also show that if these perturbation blocks overlap, we identify latents up to the smallest blocks shared across perturbations. Consequently, if there are blocks that intersect in one latent variable only, then such latents are identified up to permutation and scaling. We propose a natural estimation procedure based on this theory and illustrate it on low-dimensional synthetic and image-based experiments.

show abstract

“…More recently, Khemakhem et al (2020a) proved a major breakthrough by showing that given side information u, identifiability of the entire generative model is possible up to certain (nonlinear) equivalences. Since this pathbreaking work, many generalizations have been proposed (Hälvä and Hyvarinen, 2020;Hälvä et al, 2021;Khemakhem et al, 2020b;Li et al, 2019;Mita et al, 2021;Sorrenson et al, 2019;Yang et al, 2021;Klindt et al, 2020;Brehmer et al, 2022), all of which require some form of auxiliary information. Other approaches to identifiability include various forms of weak supervision such as contrastive learning (Zimmermann et al, 2021), group-based disentanglement (Locatello et al, 2020), and independent mechanisms (Gresele et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

“…This contrasts a recent line of work that has established fundamental new results regarding the identifiability of VAEs that requires conditioning on an auxiliary variable u that renders each latent dimension conditionally independent (Khemakhem et al, 2020a). While this result has been generalized and relaxed in several directions (Hälvä and Hyvarinen, 2020;Hälvä et al, 2021;Khemakhem et al, 2020b;Li et al, 2019;Mita et al, 2021;Sorrenson et al, 2019;Yang et al, 2021;Klindt et al, 2020;Brehmer et al, 2022), fundamentally these results still crucially rely on the side information u. We show that this is in fact unnecessary-confirming existing empirical studies (e.g Willetts and Paige, 2021;Falck et al, 2021)-and do so without sacrificing any representational capacity.…”

Section: Introductionmentioning

confidence: 98%

Identifiability of deep generative models under mixture priors without auxiliary information

Kivva¹,

Rajendran²,

Ravikumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

We prove identifiability of a broad class of deep latent variable models that (a) have universal approximation capabilities and (b) are the decoders of variational autoencoders that are commonly used in practice. Unlike existing work, our analysis does not require weak supervision, auxiliary information, or conditioning in the latent space. Recently, there has been a surge of works studying identifiability of such models. In these works, the main assumption is that along with the data, an auxiliary variable u (also known as side information) is observed as well. At the same time, several works have empirically observed that this doesn't seem to be necessary in practice. In this work, we explain this behavior by showing that for a broad class of generative (i.e. unsupervised) models with universal approximation capabilities, the side information u is not necessary: We prove identifiability of the entire generative model where we do not observe u and only observe the data x. The models we consider are tightly connected with autoencoder architectures used in practice that leverage mixture priors in the latent space and ReLU/leaky-ReLU activations in the encoder. Our main result is an identifiability hierarchy that significantly generalizes previous work and exposes how different assumptions lead to different "strengths" of identifiability. For example, our weakest result establishes (unsupervised) identifiability up to an affine transformation, which already improves existing work. It's well known that these models have universal approximation capabilities and moreover, they have been extensively used in practice to learn representations of data.

show abstract

Weakly supervised causal representation learning

Cited by 4 publications

References 11 publications

Causal Machine Learning: A Survey and Open Problems

Causal Machine Learning: A Survey and Open Problems

Weakly Supervised Representation Learning with Sparse Perturbations

Identifiability of deep generative models under mixture priors without auxiliary information

Contact Info

Product

Resources

About