An Examination of Assessment Fidelity in the Administration and Interpretation of Reading Tests

Abstract

Researchers have expressed concern about implementation fidelity in intervention research but have not extended that concern to assessment fidelity, or the extent to which pre-/posttests are administered and interpreted as intended. When studying reading interventions, data gathering heavily influences the identification of students, the curricular components delivered, and the interpretation of outcomes. However, information on assessment fidelity is rarely reported. This study examined the fidelity with which individuals paid to be testers for research purposes were directly observed administering and interpreting reading assessments for middle school students. Of 589 testing packets, 45 (8% of the total) had to be removed from the data set for significant abnormalities and another 484 (91% of the remaining packets) had correctable errors only found in double scoring. Results indicate reading assessments require extensive training, highly structured protocols, and ongoing calibration to produce reliable and valid results useful in applied research.

Examining Potential Bias in Screening Measures for Middle School Students by Special Education and Low Socioeconomic Status Subgroups

“…Accurate administration and scoring of measures, as well as accurate recording of scores, is critical in gathering efficacious data. Researchers have observed high rates of improvisation during ORF administrations coupled with counting errors in scoring (Reed & Sturges, 2012), and there have been cases of irregularities in the administration of high-stakes state tests as well (e.g., Amrein-Beardsley, Berliner, & Rideau, 2010;Tanner, 2013).…”

Common Examiner Scoring Errors on Academic Achievement Measures

“…particularly in applied research (Reed & Sturges, 2013). In a recent study on the fidelity of reading assessments within the context of school-based intervention, Reed and Sturges (2013) described the necessity for highly trained examiners and continual refinements and professional development in scoring practices for experienced as well as novice examiners.…”

Assessment Fidelity in Aphasia Research

“…Although we argue that the effects of poor assessment fidelity could negatively have an impact on interpretation and validity of research on the basis of reports of scoring errors and blinding effects in other fields (Alfonso et al, 1998;Charter et al, 2000;Cummings et al, 2014;Hrobjartsson et al, 2013;Loe et al, 2007;Poolman et al, 2007;Ramos et al, 2009;Reed & Sturges, 2013;Sherrets et al, 1979;Slate & Chick, 1989;Slate & Jones, 1990;Slate et al, 1992;Wykes et al, 2008), this has not been directly manipulated. An investigation of the relationship between effect size and assessment fidelity reporting was not undertaken due in part to the heterogeneity of the studies reviewed, uncertainty concerning whether or not assessment fidelity was monitored but not reported, and inconsistency of effect size reporting or reporting of individual data so that effect sizes could be determined.…”

“…In reading intervention research, a recent study demonstrated that 16% of the variance in student scores on a curriculum-based oral reading test could be attributed to different examiners (Cummings, Biancarosa, Schaper, & Reed, 2014). Another study reported that 8% of reading assessments had to be discarded due to significant abnormalities, and that an alarming percentage of scored assessments (> 90%) had scoring errors that were found only after double-scoring the assessments (Reed & Sturges, 2013). Several psychoeducational studies have measured examiner errors on the Wechsler Intelligence Scale for Children (WISC, WISC-Revised, WISC-Third Edition, WISC-Fourth Table 1.…”