Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates

Abstract

The Performance Assessment for California Teachers (PACT) is a high stakes summative assessment that was designed to measure pre-service teacher readiness. We examined the inter-rater reliability (IRR) of trained PACT evaluators who rated 19 candidates. As measured by Cohen’s weighted kappa, the overall IRR estimate was 0.17 (poor strength of agreement). IRR estimates ranged from −0.29 (worse than expected by chance) to 0.54 (moderate strength of agreement); all were below the standard of 0.70 for consensus agreement. Follow-up interviews of 10 evaluators revealed possible reasons we observed low IRR, such as departures from established PACT scoring protocol, and lack of, or inconsistent, use of a scoring aid document. Evaluators reported difficulties scoring the materials that candidates submitted, particularly the use of Academic Language. Cognitive Task Analysis (CTA) is suggested as a method to improve IRR in the PACT and other teacher performance assessments such as the edTPA.

Asymmetric sampling of personality (ASP): A novel approach to understanding validity limits in personality assessment and possible remedies

“…Again, significant gaps can be observed across diverse psychological dimensions, for instance, in the assessment of personality disorders (Fiedler et al, 2004) and in the context of daily behavior (Vazire & Meehl, 2008). Multiple-rater reports of a target person show the same rating discrepancies as self-other rating comparisons that can be found, for example, in the selection of educators (Lyness et al, 2021), in performance ratings (Atkins & Wood, 2006;Fleenor et al, 1996), and in medical diagnoses (Nawka & Knoerding, 2012). Regarding these rating discrepancies (i.e., self-observer and observer-observer), a prominent question in personality assessment goes mostly unnoticed: Is high inter-rater reliability (or self-peer consistency) always desirable?…”

More Evidence of Low Inter-Rater Reliability of a High-Stakes Performance Assessment of Teacher Candidates

“…Each of these five tasks includes two or three criteria (see Table 1), which are each scored on a 4-point ordinal scale, with 1 = Fail, 2 = Basic or Pass, 3 = Proficient, and 4 = Advanced. These ordinal scores are derived from the evaluation of two types of evidence in PACT submissions: artifacts (evidence that candidates submit to show teacher competence, e.g., lesson plans, videos, student work samples) and commentaries (written responses to standardized questions that provide context and rationales for the artifacts submitted) [12]. a Assessed throughout the teacher performance event.…”

“…After initial training, the evaluators were required to attend annual re-calibration events. For more details about PACT evaluator training and calibration, please see [12].…”

“…I wanted to examine the IRR in our candidates' edTPA data, but when double-scored data were requested, access was not provided because the data are proprietary to Pearson Inc. In a prior study [12], IRR was assessed in the PACT during the last academic year (2014) of its use at our school. In the current study, the sample size has been expanded by over nine times, and the sample includes PACT data that were collected during the duration of its use at our institution, from 2010 to 2015, before its replacement by the edTPA.…”

“…In the current study, the sample size has been expanded by over nine times, and the sample includes PACT data that were collected during the duration of its use at our institution, from 2010 to 2015, before its replacement by the edTPA. Because of the high-stakes nature of the TPA (receiving or not receiving your credential), examining IRR data was important for several reasons: (a) IRR data in TPAs have been reported as percent agreement without correction for chance agreement, and (b) double-scored data are not frequently reported and are even harder to obtain now that they are often proprietary to corporations such as Pearson [12].…”