Rater Agreement in Test‐to‐Curriculum Alignment Reviews

Abstract

During the development of large‐scale curricular achievement tests, recruited panels of independent subject‐matter experts use systematic judgmental methods—often collectively labeled “alignment” methods—to rate the correspondence between a given test's items and the objective statements in a particular curricular standards document. High disagreement among the expert panelists may indicate problems with training, feedback, or other steps of the alignment procedure. Existing procedural recommendations for alignment reviews have been derived largely from single‐panel research studies; support for their use during operational large‐scale test development may be limited. Synthesizing data from more than 1,000 alignment reviews of state achievement tests, this study identifies features of test–standards alignment review procedures that impact agreement about test item content. The researchers then use their meta‐regression results to propose some practical suggestions for alignment review implementation.

Rater Agreement in Alignment Studies

“…The above example demonstrates how inconsistency in alignment judgments can be a major threat to alignment studies and should not be ignored. The dependability in both the domain (.68) and standard level (.46) alignments were below the average standard‐level pairwise agreement statistic of .70 in the Traynor and Merzdorf (2018) meta‐analysis. Even though systematic bias was zero in this example, panelists were not able to consistently assign items to domains or standards.…”

“…Given this, it is troubling that alignment research is currently lacking, and there is a need for increased rigor in methods (Polikoff, this issue). Of interest here is a meta‐analysis of alignment studies on state assessments that found fewer than half of the 111 included reports provided rater agreement statistics (Traynor & Merzdorf, 2018). Rater inconsistency is a major threat to the utility of alignment studies because it is the assignment of items to standards that serves as the basis for conclusions about validity.…”

Alignment and Implications for Test Takers

“…While in the same issue, Beck (2007) cautioned educators not to “become overly data‐enchanted and attempt to wring more information about the process than is present” (p. 128). Finally, Traynor and Merzdorf (2018) discussed alignment as the correspondence between a given test's items and the objectives and noted that alignment features were not always documented consistently enough to make direct comparisons. Overall, the literature suggests different results based on different methodologies.…”