The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable

Abstract

Students in Algebra I classrooms typically learn at different rates and struggle at different points in the curriculum-a common challenge for math teachers. Cognitive Tutor Algebra I (CTA1), educational computer program, addresses such student heterogeneity via what they term "mastery learning," where students progress from one section of the curriculum to the next by demonstrating appropriate "mastery" at each stage. However, when students are unable to master a section's skills even after trying many problems, they are automatically promoted to the next section anyway. Does promotion without mastery impair the program's effectiveness?At least in certain domains, CTA1 was recently shown to improve student learning on average in a randomized effectiveness study. This paper uses student log data from that study in a continuous principal stratification model to estimate the relationship between students' potential mastery and the CTA1 treatment effect. In contrast to extant principal stratification applications, a student's propensity to master worked sections here is never directly observed. Consequently we embed an item-response model, which measures students' potential mastery, within the larger principal stratification model. We find that the tutor may, in fact, be more effective for students who are more frequently promoted (despite unsuccessfully completing sections of the material). However, since these students are distinctive in their educational strength (as well as in other respects), it remains unclear whether this enhanced effectiveness can be directly attributed to aspects of the mastery learning program.

Towards flexible personalized learning and the future educational system in the fourth industrial revolution in the wake of Covid-19

“…There are good indications that mastery techniques can be used to avoid 'teaching to the test' and avoid stress in high-stakes testing (Zimmerman & Dibenedetto, 2008) across the educational spectrum. Sales and Pane (2019) for example have started to explore intelligent tutoring and mastery learning. Anywhere-anytime tutoring on mobile devices is now practical Grant Sanderson's 3blue1brown 'lockdown math' being a good example via a YouTube channel.…”

An Aggregation Scheme for Increased Power

“…In underlying concept if not in its goals, the method relates to instrumental variables estimation (Bloom, 1984;Angrist et al, 1996;Baiocchi et al, 2014) and principal stratification (Frangakis and Rubin, 2002;Page, 2012;Sales and Pane, 2019). But whereas Sales and Pane (2021), for example, use principal stratification to estimate separate effects for latent subgroups distinguished in terms of dosage level, we marshal related considerations to inform aggregation of effects across manifest subgroups receiving or likely to receive differing doses.…”

Polytomous IRT Models With A Cut-Point Formation Mechanism: An Application To The Measurement of Anxiety

“…IRT models have a rich history (Bock, 1997;Fisher and Molenaar, 1997), starting with the work by Thurstone (1925), Rasch (1960), and Lord (1980). They have found applications in education (Sales and Pane, 2019;Schofield, 2015;Wang, Berger and Burdick, 2013), public policy (Treier and Jackman, 2008), demography (McParland et al, 2014), business (de Jong, Steenkamp and Fox, 2007), cross-cultural research (de Jong and Steenkamp, 2010), and clinical psychology (Reise and Waller, 2009), where they have been used to measure a variety of abilities, attitudes, traits, and mental states (Reise et al, 2021;Shea, Tennant and Pallant, 2009). For over twenty years, IRT models have been formulated in the Bayesian framework, and estimated using Markov Chain Monte Carlo (MCMC) methods (Béguin and Glas, 2001;Bradlow and Zaslavsky, 1999;Li and Baser, 2012;Patz et al, 2002;Wang, Bradlow and Wainer, 2002).…”