Rater Agreement in Alignment Studies

Fina, Anthony D.

doi:10.1111/emip.12338

Cited by 1 publication

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The use of item developer alignments can assist auditors in determining if test forms were created as intended; whereas, the use of independent teacher panels can provide evidence that the alignments are generalizable. However, there are complexities for alignment, among which are the following: (a) standards can be interpreted in different ways and are written at different grain sizes across states, (b) rater consensus is difficult and time consuming to achieve so decision rules are often adopted regarding the proportion of agreement within a panel needed to consider the item aligned, and (c) the greater the number of alignments to be made, the lower the agreement statistics will be (Davis‐Becker & Buckendahl, 2013; Fina, 2020; Webb et al., 2007).…”

Section: Alignment To Score Interpretationsmentioning

confidence: 99%

The Relationship between Item Developer Alignment of Items to Range Achievement‐Level Descriptors and Item Difficulty: Implications for Validating Intended Score Interpretations

Schneider

Agrimson²,

Veazey³

2021

Educational Measurement

View full text Add to dashboard Cite

This paper presents results of a score interpretation study for a computer adaptive mathematics assessment. The study purpose was to test the efficacy of item developers’ alignment of items to Range Achievement‐Level Descriptors (RALDs; Egan et al.) against the empirical achievement‐level alignment of items to investigate the use of RALDs as the epicenter of the test score interpretation validity argument. Item developers aligned 82%–87% of items in a computer adaptive item bank for Grades 3–8 mathematics to the assessment's RALDs that had been reconciled to the test scale after standard setting. The degree of consistency between the hypothesized alignment and actual alignment was examined using agreement statistics. Item developers correctly identified the empirical achievement level of 56%–60% of the items, which was above the chance level of agreement. An emerging technique known as embedded standard setting (ESS; Lewis and Cook) was then used to evaluate whether score interpretations based on item developer classifications of items to RALDs were comparable to the score interpretations derived from the cut scores set in 2018. Score interpretations for the first two achievement levels were consistent across administrations; however, for the most advanced students score interpretations were not maintained.

show abstract

Section: Alignment To Score Interpretationsmentioning

confidence: 99%