Intervention and Identifiability in Latent Variable Modelling

Abstract

We consider the use of interventions for resolving a problem of unidentified statistical models. The leading examples are from latent variable modelling, an influential statistical tool in the social sciences. We first explain the problem of statistical identifiability and contrast it with the identifiability of causal models. We then draw a parallel between the latent variable models and Bayesian networks with hidden nodes. This allows us to clarify the use of interventions for dealing with unidentified statistical models. We end by discussing the philosophical and methodological import of our result.

Disentangling Model Multiplicity in Deep Learning

“…In learning rate regime, we used batch size 64 while using the learning rates [0.003, 0.002, 0.001, 0.0003, 0.0001, 0.00003, 0.00001]. In batch size regime, we used learning rate 0.0001 and batch sizes [8,16,32,64,128,256,512]. Cross entropy loss was used, with ADAM optimizer (β 1 = 0.9, β 2 = 0.999, = 1e − 08).…”

“…Many prior works approach the phenomenon via elimination or reduction to uncertainty. The elimination of the problem can be defended if one can plausibly unearth the underlying causal properties via designing a set of interventions to that end (see D'Amour et al [1] for a large scale empirical evaluation, Romeijn and Williamson [8] for theoretical analysis). Yet, this rarely happens in ML modeling except in the very theory of causal inference itself (e.g.…”

Latent Causal Invariant Model

“…These are supported by the framework of generative model ; Khemakhem, Monti, Kingma and Hyvärinen (2020); Kingma and Welling (2014); Suter et al (2019) which has natural connection with the causal graph Schölkopf (2019) that the edge in the causal graph reflects both the causal effect and also the generating process. Until now, perhaps the most similar work to us are Romeijn and Williamson (2018) and Teshima et al (2020) which also need multiple training domains and get access to a few samples in the target domain. Both work assumes the similar causal graph with us but unlike our LaCIM, they do not separate the latent factors which can not explain the spurious correlation learned by supervised learning Ilse et al (2020).…”

“…Both work assumes the similar causal graph with us but unlike our LaCIM, they do not separate the latent factors which can not explain the spurious correlation learned by supervised learning Ilse et al (2020). Besides, the multiple training datasets in Romeijn and Williamson (2018) refer to intervened data which may hard to obtain in some applications. We have verified in our experiments that explicitly disentangle the latent variables into two parts can result in better OOD prediction power than mixing them together.…”

Towards a Logical Framework for Latent Variable Modelling

“…Several conceptual concerns are considered in the literature, including as examples questions of causality (Cliff, 1983), or the relevance or otherwise of correlations and covariances calculated on betweensubjects data to any single individual (Borsboom, 2005;Weinberger, 2015). Further problems that remain for applications of the latent variable model that are not dealt with in this paper include decisions regarding calculation techniques for the parameters of the latent variable model, measurement invariance (Meredith, 1993), unidimensionality (McDonald, 1999, model identification (Romeijn & Williamson, 2018), and the problem of equivalent models (Maccallum et al, 1993).…”