Mastery measurement in mathematics and the goldilocks effect.

Abstract

Math curriculum-based measurement (CBM) is an essential tool for multi-tiered systems of support decision making, but the reliability of math CBMs has received little research, particularly using more rigorous methods such as generalizability (G) theory. Math CBM is historically organized into two domains: mastery measures and general outcome measures. This paper details 17 concurrent G and dependability studies in a partially crossed design investigating the reliability of mastery measure CBMs for students (N = 263) in Grades K, 1, 3, 5, and 7. This study extends prior research by including novel grade levels and more rigorous math content; using generated rather than static measures; embedding a replication; examining bias by race and sex; and evaluating a simpler scoring method of answers correct as compared to digits correct. Most of the variance in scores was accounted for by student. Probe form effects accounted for less than 5% of the variance for 16 of 17 measures and results replicated across days. G coefficients exceeded .75 on the first trial for 14 of 17 measures. G studies were repeated by race, sex, and scoring metric. Overall, 1–4 min of assessment was sufficient to meet reliability thresholds, which exceeds prior findings for general outcome measures. This study supports the reliability of mastery measurement in math CBM and as a precise tool to be used in the screening process.

Reliability of Computer-Based CBMs Versus Paper/Pencil Administration for Fact and Complex Operations in Mathematics

“…Instability of estimates would have indicated a need for a larger sample. Variance estimates were remarkably consistent between the precursor study (Solomon et al, 2022) and this study. In addition, variance estimates were stable across the four one-facet studies, indicating that the sample size was sufficient.…”

“…G studies in school psychology have included sample sizes ranging as low as 9 students (Volpe & Briesch, 2016) to about 50 students per grade level examined (Solomon et al, 2022). Reliability estimates from a precursor study that used G-theory to examine the measures used in this study informed our sample size (Solomon et al, 2022). The precursor study examined reliability of 17 measures used for screening across Grades K, 1, 3, 5, and 7 using G-theory.…”

“…Both measures were scored according to publisher guidelines for answers correct (as opposed to digits correct) and all technical adequacy data provided below used answers correct. A recent study examined reliability differences for digits correct and answers correct and found that answers correct was slightly superior to digits correct in reliability (Solomon et al, 2022).…”

“…Twenty-one students in two third-grade classes from the same school participated in this study. G studies in school psychology have included sample sizes ranging as low as 9 students (Volpe & Briesch, 2016) to about 50 students per grade level examined (Solomon et al, 2022). Reliability estimates from a precursor study that used G-theory to examine the measures used in this study informed our sample size (Solomon et al, 2022).…”

“…Later studies used G-theory to examine stratified content arrangement versus random content arrangement on math measures, finding that stratified content resulted in more generalizable and dependable scores (Christ & Vining, 2006; Methe et al, 2015). More recently, G-theory has been used to examine math CBM using generated alternate-form measures of specific skills finding that such measures could be used reliably for decision-making in most cases with only a single probe administration (Solomon et al, 2022). Each of these studies informed math CBM in terms of the content sampled and the number of probes administered to yield reliable scores for instructional decision-making.…”