School Psychology Trainees’ Administration and Scoring Errors on the Woodcock-Johnson IV Tests of Achievement

Abstract

Sixty-three Woodcock–Johnson IV Tests of Achievement protocols, administered by 26 school psychology trainees, were examined to determine the frequency of examiner errors. Errors were noted on all protocols and ranged from 8 to 150 per administration. Critical (e.g., start, stop, and calculation) errors were noted on roughly 97% of protocols. Wilcoxon signed-rank tests indicated multiple subtests were more prone to both critical and non-critical (e.g., failure to record answers verbatim, failure to record qualitative observations) errors; critical errors were generally more common on subtests with objective scoring criteria (i.e., Written Expression and Spelling) and non-critical errors were more frequently observed on subtests that required the recording of answers verbatim. Based on these findings, we encourage trainers to place increased scrutiny on trainee’s objective scoring performance and on requiring recording responses verbatim. Areas of needed future research studies are also discussed.

Administration and Scoring Errors on the Woodcock–Johnson IV Tests of Achievement: Before and During COVID-19

“…We concur with Lockwood et al (2023) that the subjective scoring rules of Writing Samples and Spelling likely contribute to the greater tendency towards critical errors on these subtests. However, like Lockwood and colleagues, we also found that Calculation was more prone to critical errors though it has objective scoring rules.…”

“…Like Lockwood and colleagues, we found failures to record verbatim on every protocol (M = 29.98); this provides confirmatory evidence that from early in their training, test administrators are not consistently collecting these data. Similar to Lockwood et al (2023), we found critical errors on nearly every protocol that we examined; both studies also reported a mean of approximately 15 critical errors per protocol. Additionally, both studies found critical errors were most noted on Calculations, Spelling, and Writing Samples, with at least one error found on the majority of protocols.…”

“…Like Lockwood et al (2023), we also found that Word Attack, Letter-Word Identification, Passage Comprehension, and Applied Problems were most prone to non-critical errors. Like these researchers, we also found a mean of 5-10 errors per subtest per protocol.…”

“…Data coding was completed by two school psychology doctoral students who had completed at least two courses on NRT administration and at least one full year of school psychology practicum. A rubric for coding examiner errors was adopted from a previous study (i.e., Lockwood et al, 2023). The initial percentage of agreement between coders for interrater reliability was 95.6%.…”

“…Administration and scoring errors on NRTs of cognitive ability has been the subject of dozens of publications (e.g., Alfonso et al, 1998;Lobello, & Holley, 1999;Loe 2014;Mrazik et al, 2012;Oak et al, 2019;Sherrets et al, 1979;Slate et al, 1991Slate et al, , 1993. These studies suggest that administration and scoring errors on NRTs of cognitive ability are ubiquitous (Lockwood et al, 2023). Furthermore, these errors are often consequential, affecting the overall score on nearly a quarter of protocols (see Styck & Walsh, 2016, for meta-analysis of errors on the Wechsler series).…”