Causal Inference and Bias in Learning Analytics

Abstract

As a research field geared toward understanding and improving learning, Learning Analytics (LA) must be able to provide empirical support for causal claims. However, as a highly applied field, tightly controlled randomized experiments are not always feasible nor desirable. Instead, researchers often rely on observational data, based on which they may be reluctant to draw causal inferences. The past decades have seen much progress concerning causal inference in the absence of experimental data. This paper introduces directed acyclic graphs (DAGs), an increasingly popular tool to visually determine the validity of causal claims. Based on this, three basic pitfalls are outlined: confounding bias, overcontrol bias, and collider bias. Further, the paper shows how these pitfalls may be present in the published LA literature alongside possible remedies. Finally, this approach is discussed in light of practical constraints and the need for theoretical development.

Using causal models to bridge the divide between big data and educational theory

“…Elsewhere, we have argued in more detail that DAGs, from informal sketches to more rigorous models, offer visual affordances that scaffold conversations between data and educational experts, comparable to the way that concept maps and other diagramming schemes relieve cognitive load by providing a form of shared, persistent, but malleable, external memory aid (Hicks et al, 2022). Similarly, the recent work by Weidlich et al (2022) demonstrates how causal models can help modellers and educators to work together to identify sources of bias in various models, and ways in which they might be ameliorated. In attempting to construct a DAG we will sometimes construct competing models that are empirically testable, and these can be interrogated and challenged by educational experts.…”

“…This process of identifying variable sets from the causal structure in order to estimate a causal effect is known as identification . Being able to focus on identification separately to the estimation of causal effects is a distinguishing feature of the DAG based approach (Weidlich et al, 2022). While most work with causal models has focused on identification and estimation of causal effects, or the discovery of causal structures from data, this paper will explore the affordances of the implied conditional independence relationships of a causal DAG for the development of a stronger (ie, more formalised) theory.…”

“…Importantly, causal modelling provides a way to move beyond the 'correlation does not imply causation' mantra that is common in many fields using observational data, without the need to resort to the running of RCTs. Recent papers in LA have suggested that this opens up opportunities to involve stakeholders in the construction of educational models (Hicks et al, 2022) and to reduce potential bias in educational research that relies upon observational data (Weidlich et al, 2022).…”

“…A collider, also recently called an 'inverted fork' by Weidlich et al (2022), describes the pattern A → B ← C. We would not expect to see an association arise between the variables A and C if we had theorised a pattern of this form. However, in this case we would expect to see an association arise if we were to control for B.…”

Aligning the Goals of Learning Analytics with its Research Scholarship

“…Importantly, causal modelling can be employed in all phases of research, e.g., study planning, analysis, and literature appraisal (Weidlich et al, 2023). In the absence of research contexts amenable to rigorous experimental protocols, DAGs can also provide a principled approach to thinking about and understanding sources of bias in a variety of research designs in LA (Weidlich et al, 2022). For a deeper introduction to causal DAGs and their potential applications in LA see Weidlich et al (2022) or Hicks et al (2022).…”