Inter‐rater Reliability of the Cloze Reading Inventory as a Qualitative Measure of Reading Comprehension

Abstract

The study reported herein was conducted to ascertain the nature of potential between-rater subjectivity in scoring quantitative and qualitative cloze based assessments of reading comprehension. Fifty students were randomly chosen from a group of 456 students used in the standardization of the Cloze Reading Inventory (CRI). As each student took four different forms of the CRI, there were 200 passages each rated by three of seven raters for a total of 600 passages used to determine interrater reliability. The seven raters each had a minimum of five years teaching experience, were from the elementary or secondary level, and were graduate students in a reading specialist program. The intraclass correlation method was employed to derive coefficients from the mean score ratings of the raters for each of the passages, each of the four interpretive values, and each of grades 3, 5, 7, 9, and 11. The results of the repeated measures analysis of variance indicated that interrater reliabilities were generally consistently high across passages, interpretive values, and grade levels.

Interrater reliability of constructed response items in standardized tests of reading

“…The study of interrater reliability of reading test items is a limited area of research, and the extent of reliability, as well as the exact definition of what might qualify as a "high level" of reliability, will depend on both item construction and on the level of rater training (DeSanti & Sullivan, 1984;Taboada, Tonks, Wigfield & Guthrie, 2013). Therefore, any test program that requires subjective scoring needs to evaluate and validate their own proportion of rater reliability (Bejar, 2012).…”

Samstämmighet i lärares bedömning av nationella prov i läsförståelse

“…Studier av interbedö marreliabilitet av läsfö rmåga är ett begränsat forskningsområde, men några undersö kningar har visat att det går att uppnå hö ga nivåer av samstämmighet i standardiserade provsystem (DeSanti & Sullivan 1984). Däremot rapporterar Illinois State Board of Education (2013) att vid genomfö rande av SATtesterna 1 i läsning 2013 låg nivån fö r exakt överensstämmelse på moderata 65Á67 % vid bedö mning av ''extended-response items'', där bedö marna bedö mde varje elevsvar på en skala från 0Á4.…”