The Role of A/B Tests in the Study of Large-Scale Online Learning

Savi, Alexander Olof; Williams, Joseph Jay; Maris, Gunter; Maas, Han L. J. van der

doi:10.31219/osf.io/83jsg

Cited by 8 publications

(7 citation statements)

References 18 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the experiment was double-blinded, a condition that is often difficult to satisfy in educational research. Savi et al (2017) explore the various benefits and challenges of A/B tests in more detail.…”

Section: Experimental Designmentioning

confidence: 99%

Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Savi¹,

C²,

Cornelisz³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Effort is key in learning, evidenced by its omnipresence in both empirical findings and educational theories. At the same time, students are consistently found to avoid effort. In this study, we investigate whether limiting effort avoidance improves learning outcomes, and explore for whom this would be the case. In a large-scale computer adaptive practice system for primary education, over 150,000 participants were distributed across four conditions in which a problem-skipping option was delayed for 0, 3, 6, or 9 seconds. The results show that after a 14 week period, no average treatment effects in learning outcomes can be found between conditions. A substantive typology of students, based on the expected target mechanisms of the intervention, neither shows consistent conditional average treatment effects. Nevertheless, the substantive typology is shown to be meaningful, as the different types—toilers, skippers, and rushers—differ with respect to their learning outcomes. We argue that although the scale of the experiment suggests a precise null finding, the cumulative nature of the effect of problem skipping cautions against generalizing this finding to sustained intervention.

show abstract

Section: Experimental Designmentioning

confidence: 99%

Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Savi¹,

C²,

Cornelisz³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…An experiment driven strategy is an iterative testing strategy and allows for evaluating learning interventions and reveal pattern and side-effects [18]. Online experiments, or A/B tests, determine what is needed to attain a certain knowledge level or domain rating.…”

Section: Governing Strategiesmentioning

confidence: 99%

Governing games Adaptive game selection in the Math Garden

2020

View full text Add to dashboard Cite

Utilizing online digital educational content has become the norm when teaching young students. A variety of adaptive educational practice sys- tems is readily available and allows students to practice various domains, on a preferred difficulty and pace. However, due to the intensification of the teaching profession and the possibilities of practicing from home, students might be left unsupervised, and as a result do not practice domains that are most important. This study proposes a solution to govern these students, i.e., provide comput- erized data driven supervision that guides students in practicing domains most important with no intervention of a teacher. Through an experiment involving 13 578 participants, a new governing method was tested and found to have positive effects on both engagement and learning, with almost no changes to the visual interface needed. Governing seems a promising technique in general, and was effectively tested and introduced in Math Garden.

show abstract

“…As can be seen, the level of conclusions here is not the individual learner, but rather the general inferences that can be made about the relationship of explanatory variables across the learners and the items. Therefore, the explanatory approach might be very useful in A/B tests (large-scale online randomized controlled experiments) which seek the optimal way to organize learning experience within the digital environment, for instance, in comparing two types of video lectures -produced in professional studio (A) and hand-crafted (B) -by their effect on learners' performance in a MOOC (Kizilcec & Brooks, 2017;Savi, Williams, Maris, & van der Maas, 2017).…”

Section: Explanatory Modelingmentioning

confidence: 99%

Psychometrics of MOOCs: Measuring Learners’ Proficiency

Abbakumov

Desmet

Noortgate

2020

Psychologica Belgica

View full text Add to dashboard Cite

Massive open online courses (MOOCs) are "one of the most significant technological developments in higher education in the past decade" (Deng, Benckendorff, & Gannaway, 2019, p. 48). MOOCs are large scale web-based courses developed by universities, solely or in cooperation with industrial partners (for example, Coursera, 2015), in which anyone with internet access can participate. MOOCs are published on provider platforms, for instance, Coursera, edX, XuetangX, FutureLearn, Udacity, MiriadaX. MOOCs are a relatively new instructional form -the term did not exist before 2008 (Major & Blackmon, 2016). However, their popularity grows fast -over a decade more than 800 universities offered to learners more than 9,400 MOOCs (Shah,

show abstract

The Role of A/B Tests in the Study of Large-Scale Online Learning

Cited by 8 publications

References 18 publications

Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Governing games Adaptive game selection in the Math Garden

Psychometrics of MOOCs: Measuring Learners’ Proficiency

Contact Info

Product

Resources

About