Decision-Making Accuracy of CBM Progress-Monitoring Data

Abstract

This study examined the diagnostic accuracy associated with decision making as is typically conducted with curriculum-based measurement (CBM) approaches to progress monitoring. Using previously published estimates of the standard errors of estimate associated with CBM, 20,000 progress-monitoring data sets were simulated to model student reading growth of twoword increase per week across 15 consecutive weeks. Results indicated that an unacceptably high proportion of cases were falsely identified as nonresponsive to intervention when a common 4-point decision rule was applied, under the context of typical levels of probe reliability. As reliability and stringency of the decision-making rule increased, such errors decreased. Findings are particularly relevant to those who use a multi-tiered response-to-intervention model for evaluating formative changes associated with instructional intervention and evaluating responsiveness to intervention across multiple tiers of intervention. Keywords assessment of interventions/outcomes, diagnostic classification models, measurement, response to intervention (RTI)/multi-tiered system of supports (MTSS), statistical analyses, measurement Curriculum-based measurement (CBM) is a set of standardized and specific measurement procedures that can be used to index student performance in the basic skill areas of early literacy and reading, early numeracy and mathematics computation and application, spelling, and written expression (

Assessing the consequential validity of early literacy progress monitoring data: An investigation of the accuracy of decision rules to evaluate response to instruction.

“…One key similarity between the results observed in this study and CBM-R decision-rule research was the tendency for recommendations from the data-point rule to suggest that an intervention should not be changed (Hintze et al, 2018; Van Norman & Christ, 2016a; Van Norman & Parker, 2018). This is particularly concerning, as the tendency in educational screening contexts is to err on the side of overidentifying students to minimize the chance that at-risk students do not receive intervention (Kilgus et al, 2014).…”

“…Van Norman and Parker (2018) found such poor agreement rates between the three-point rule and end-of-year performance that they suggested the rule be retired in its current form. Using four or five data points rather than three did not appear to improve the accuracy of recommendations (Hintze, Wells, Marcotte, & Solomon, 2018). The trend-line rule has shown some promise when appropriate conditions are met.…”

The effect of nonlinear growth on the accuracy of curriculum-based measurement of reading decision rules.

“…Systematic lines of research have begun to emerge that have explored various influences on the accuracy of treatment recommendations from decision rules. In general, researchers have found that the accuracy of recommendations improve if (a) educators use the median or trend line rule as opposed to the data point rule, (b) commercial probe sets and standardized directions are used to minimize measurement error, (c) at least one observation is collected per week for at least 10–12 weeks before applying a rule, (d) multiple WRCM scores are collected at each measurement occasion, and (e) observed growth is highly discrepant from the goal line (Christ, Zopluoglu, Monaghen, & Van Norman, 2013; Hintze, Wells, Marcotte, & Solomon, 2018; Van Norman & Christ, 2016; Van Norman, Christ, & Newell, 2017; Van Norman et al, 2018). It is worth nothing that all studies reviewed thus far have assumed that goal lines, as well as student growth, were monotonic and linear across the school year.…”

Curriculum Based Measurement Maze: A Review

“…Deno (1985) highlighted at an early stage the need to create easy-to-use and reliable tools for measuring and monitoring students' progress, which were mainly based on the informal observations of their teachers. In this context, the CBM measurement can be used both in general and special education, aiming universal screening in order to identify students at risk and evaluate intervention programs (Deno & Fuchs, 1987;Hintze, 2009;Hintze, Wells, Marcotte, & Solomon, 2018).…”