Trying to Outrun Causality with Machine Learning: Limitations of Model Explainability Techniques for Identifying Predictive Variables

Abstract

Machine Learning explainability techniques have been proposed as a means of 'explaining' or interrogating a model in order to understand why a particular decision or prediction has been made. Such an ability is especially important at a time when machine learning is being used to automate decision processes which concern sensitive factors and legal outcomes. Indeed, it is even a requirement according to EU law. Furthermore, researchers concerned with imposing overly restrictive functional form (e.g., as would be the case in a linear regression) may be motivated to use machine learning algorithms in conjunction with explainability techniques, as part of exploratory research, with the goal of identifying important variables which are associated with an outcome of interest. For example, epidemiologists might be interested in identifying 'risk factors' -i.e. factors which affect recovery from disease -by using random forests and assessing variable relevance using importance measures. However, and as we demonstrate, machine learning algorithms are not as flexible as they might seem, and are instead incredibly sensitive to the underling causal structure in the data. The consequences of this are that predictors which are, in fact, critical to a causal system and highly correlated with the outcome, may nonetheless be deemed by explainability techniques to be unrelated/unimportant/unpredictive of the outcome. Rather than this being a limitation of explainability techniques per se, we show that it is rather a consequence of the mathematical implications of regression, and the interaction of these implications with the associated conditional independencies of the underlying causal structure. We provide some alternative recommendations for researchers wanting to explore the data for important variables.

Prespecification of Structure for Optimizing Data Collection and Research Transparency by Leveraging Conditional Independencies

“…Running the regression E[C|B, A] from data generated according to a fully mediated DGP will result in the same consequences as above: the fact we have included B means that the importance given to A will be zero (notwithstanding finite sample deviations). Clearly, therefore, an understanding of the structure is therefore absolutely crucial for constructing the regression models (Vowels, 2022). For instance, if A is a treatment variable and we do not recognise B as a mediator, the inclusion of B in the model will result in a negligible coefficient estimate for A which may well mislead us to think the treatment is ineffective.…”