Embedding Experiments: Staking Causal Inference in Authentic Educational Contexts

Abstract

To identify the ways teachers and educational systems can improve learning, researchers need to make causal inferences. Analyses of existing datasets play an important role in detecting causal patterns, but conducting experiments also plays an indispensable role in this research. In this article, we advocate for experiments to be embedded in real educational contexts, allowing researchers to test whether interventions such as a learning activity, new technology, or advising strategy elicit reliable improvements in authentic student behaviours and educational outcomes. Embedded experiments, wherein theoretically relevant variables are systematically manipulated in real learning contexts, carry strong benefits for making causal inferences, particularly when allied with the data-rich resources of contemporary e-learning environments. Toward this goal, we offer a field guide to embedded experimentation, reviewing experimental design choices, addressing ethical concerns, discussing the importance of involving teachers, and reviewing how interventions can be deployed in a variety of contexts, at a range of scales. Causal inference is a critical component of a field that aims to improve student learning; including experimentation alongside analyses of existing data in learning analytics is the most compelling way to test causal claims.

Primary pupils, science and a model bird species: Evidence for the efficacy of extracurricular science education

“…At this point it is relevant to recall that the two schools differed in an additional aspect, i. e. in school B, science education was taught by the same teacher in both the project and the control group, whereas in school A, different teachers were responsible for science education in the project and the control group. In future investigations, more attention will be paid to the random choice of project and control groups, particularly when the same teacher is involved in both, as suggested by Motz et al [30]. Alternatively, including a third school where no intervention was carried out would add validity as a further control group.…”

“…The impact of a teacher’s guidance on students, particularly in enquiry-based learning was also highlighted by Minner et al [8] and Decristan et al [5]. Potvin & Hasni [33] emphasized that the educators’ targeted efforts also affect the students’ interest, motivation and attitudes to science, although not necessarily test performance [30]. The results of the present study indicate that by fostering teachers’ attitudes toward science this may influence and improve their students’ interest and attitudes, which may subsequently foster their learning achievements.…”

What Does Methodology Mean for Learning Analytics?

“…Learning gains might actually be diminished in comparison to non-adaptive alternatives. There are different ways to evaluate the design of an adaptive system; for example, one may use experimental designs (see, for example, Motz, Carvalho, de Leeuw, & Goldstone [2018], in this issue). Alternatively, when a measurement model exists, then a type of internal verification arises from model fit analysis.…”

ManyClasses 1: Assessing the generalizable effect of immediate versus delayed feedback across many college classes

“…Unfortunately, these applied educational experiments are often conducted in very narrow contexts (e.g., in single classrooms on a single topic with researchers who vigilantly curate and monitor the study to ensure compliance); thus, it can be easy to dismiss conflicting results as idiosyncratic to the specific context. Yet, these idiosyncratic educational contexts are precisely those that psychological research aims to improve (Koedinger, Booth, & Klahr, 2013;Motz, Carvalho, de Leeuw, & Goldstone, 2018). To provide legitimate estimates of the benefits of recommended practices in authentic educational environments, we need rigorous, experimental research that extends beyond the bounds of any single class.…”

“…The model estimated that an increase of 1SD in course grade was associated with a 0.039 to 0.071 (95% HDI) decrease in observed Δz score in the non-incentivized classes, and a 0.032 to 0.064 (95% HDI) decrease in observed Δz score in the incentivized classes; students who performed well also tended to be the students who showed a relative benefit of delayed feedback. [Add 2 to 3 paragraphs discussing these findings as they relate to the feedback literature, the literature on evidence-based educational practices, and on the application of cognitive science to education more broadly, which includes a consideration of the ethics of embedding experiments in real classes (Motz et al, 2018). ]…”