A Proposed Framework for Evaluating Alignment Studies

Abstract

Evaluating the multiple characteristics of alignment has taken a prominent role in educational assessment and accountability systems given its attention in the No Child Left Behind legislation (NCLB). Leading to this rise in popularity, alignment methodologies that examined relationships among curriculum, academic content standards, instruction, and assessments were proposed as strategies to evaluate evidence of the intended uses and interpretations of test scores. In this article, we propose a framework for evaluating alignment studies based on similar concepts that have been recommended for standard setting (Kane). This framework provides guidance to practitioners about how to identify sources of validity evidence for an alignment study and make judgments about the strength of the evidence that may impact the interpretation of the results.

Alignment of Teacher-Developed Curricula and National Standards in Qatar’s National Education Reform

“…Teachers in this context are given significant roles in developing and aligning curriculum to standards without having the skill, professional development or time to do so effectively. As mentioned by Davis-Becker and Buckendahl (2013), developing and aligning standards requires a great deal of professional development and experience. These teachers were at the same time being faced with new teaching methods and learning how to deal with standards.…”

“…7, No. 10; A number of alignment strategies have been reviewed by Davis-Becker and Buckendahl (2013). They cite Webb's (1999) alignment strategy that evaluates assessment items aligned with content standards.…”

Rater Agreement in Test‐to‐Curriculum Alignment Reviews

“…Commonly‐used alignment methods require panels of subject‐matter experts (SMEs) to rate alignment between test items and particular objective statements in a curricular standards document, to match items to objectives to which they correspond, or to independently match both items and objectives to a content classification table (Flowers, Wakeman, Browder, & Karvonen, ; Porter, Polikoff, Zeidner, & Smithson, ; N. L. Webb, Alt, Ely, Cormier, & Vesperman, ). Prior to judging test item‐to‐objective alignment, a facilitator typically guides SME panelists through some training, which may include, for instance, practice coding sample objectives or items (Davis‐Becker & Buckendahl, ). The panelists are then instructed to code document content independently of one another.…”

“…However, if alignment ratings indicate high disagreement among SMEs that suggests serious random or partially systematic error during rating, the alignment results may become “a function of who does the rating” rather than reflecting test items’ content features (N. M. Webb, Herman, & Webb, , p. 25). Failure to resolve or account for this disagreement in any way may be problematic for meaningful use of alignment results during test revision (Davis‐Becker & Buckendahl, ).…”

“…Although careful selection of qualified SMEs who have knowledge of the relevant curricular standards should reduce the potential for rater misinterpretation of objectives or test items, alignment ratings (or matches) made by individual judges may be influenced by random error or systematic bias. Observed absolute disagreement in item alignment ratings among SME reviewers has been attributed to multiple possible sources: (1) some objective statements in particular standards documents may be too vague to interpret with confidence (e.g., Resnick et al., ), (2) some items may be correctly answered using multiple distinct solution paths or lines of reasoning, which may produce uncertainty about what objective(s) they measure (Leighton & Gokiert, ), (3) SMEs may misunderstand the alignment rating criteria due to insufficient training (Davis‐Becker & Buckendahl, ), (4) SMEs may seriously misinterpret particular items or objectives (D'Agostino et al., ), (5) SMEs may be too strict or lenient, tending to find too many or few item‐objective matches, or to assign higher or lower ratings than warranted by the tasks’ content (e.g., Roach, Niebling, & Kurz, ), (6) SMEs may lack sufficient content expertise (N. M. Webb et al., ), or vary in their understanding of the “developmental levels and prior instructional experience” of the test‐taker population (Herman, Webb, & Zuniga, , p. 121), either of which may lead to ill‐informed judgments about test items’ cognitive requirements, and (7) SMEs may make physical coding errors (N. M. Webb et al., ), or (8) SMEs may give error‐prone ratings due to fatigue (Sireci, ).…”