Diagnostic Accuracy of Multivariate Universal Screening Procedures for Reading in Upper Elementary Grades

Abstract

We examined the diagnostic accuracy and efficiency of three approaches to universal screening for reading difficulties using retrospective data from 1,307 students in Grades 3 through 5. School staff collected screening data using the Measures of Academic Progress (MAP), a curriculum-based measure (CBM), and running records (RR). The criterion measure was a high-stakes state accountability test aligned with the Common Core State Standards. We examined the diagnostic accuracy of the tests in isolation, as multivariate batteries, and via a simulated gated-screening approach. CBM and RR data resulted in unacceptable diagnostic accuracy across all three grades. In the fourth grade, the MAP alone resulted in the best balance of sensitivity and specificity. Among third- and fifth-grade students, the multivariate combination of MAP and CBM demonstrated the best balance between diagnostic accuracy and efficiency. Gated-screening increased specificity but lowered sensitivity. Results highlight the need for population-specific considerations in universal screening.

Evaluating the Criterion Validity and Classification Accuracy of Universal Screening Measures in Reading

“…Next, hierarchical linear regression was used to determine the shared and unique variance of screening measures in relation to the Georgia Milestones. MAP was entered first because it shares the characteristic of being a computer-administered assessment, is convenient to employ, is supported by an emerging peer-reviewed literature base (e.g., Ball & O'Connor, 2016;January & Ardoin, 2015;Klingbeil et al, 2017), and has content overlap with the Georgia Milestones.…”

“…Strong correlations (r = .77) between fall and spring MAP were reported, in addition to sensitivity and specificity levels of .85 and .81, respectively, using test-developer benchmarks. In a recent study using publisher-recommended cut scores, Klingbeil, Nelson, Van Norman, and Birr (2017) found strong correlations (r = .74) between fall MAP scores and a spring-administered state assessment for students in Grade 3.…”

“…Test-developer suggested benchmarks may or may not be tailored to fit the needs of a specific school or population of students. As an alternative, locally derived cutscores from Receiver Operating Characteristic (ROC) curve analyses may be used to obtain accurate predictions of student performance (Klingbeil, Nelson, Van Norman, & Birr, 2017;Walker, Small, Severson, Seeley, & Feil, 2014). The area under the curve (AUC) value obtained from ROC curve analyses represents the overall accuracy of a screening measure ranging in value from no predictive value, 0.0, to perfect accuracy, 1.0 (Christ & Nelson, 2014;Walker et al 2014).…”

“…The area under the curve (AUC) value obtained from ROC curve analyses represents the overall accuracy of a screening measure ranging in value from no predictive value, 0.0, to perfect accuracy, 1.0 (Christ & Nelson, 2014;Walker et al 2014). While maintaining a balance between levels of sensitivity and specificity and reaching acceptable AUC values, it is important that benchmarks are contextually appropriate regardless of the methods used to create them (Klingbeil et al, 2017;Nelson, Van Norman, & Lackner, 2017).…”

Accuracy and completeness are required: A response to Dreher et al (2019)

“…An expanded discussion of the points above was published before our 2008 paper but it does not appear to have been reviewed by Dreher et al Furthermore, any suggestion that the approach I (DS) took in 2008 is illogical is refuted by the widespread use of classification accuracy analyses (ROC curves, positive predictive value, etc.) in educational, behavioral, and clinical contexts . Indeed, the Center on Response to Interventions and the National Center on Intensive Intervention have both established diagnostic accuracy indices as the only method used to evaluate the accuracy of classification decisions made by educational tests.…”