Implications of Evidence‐Centered Design for Educational Testing

Abstract

Evidence‐centered assessment design (ECD) provides language, concepts, and knowledge representations for designing and delivering educational assessments, all organized around the evidentiary argument an assessment is meant to embody. This article describes ECD in terms of layers for analyzing domains, laying out arguments, creating schemas for operational elements such as tasks and measurement models, implementing the assessment, and carrying out the operational processes. We argue that this framework helps designers take advantage of developments from measurement, technology, cognitive psychology, and learning in the domains. Examples of ECD tools and applications are drawn from the Principled Assessment Design for Inquiry (PADI) project. Attention is given to implications for large‐scale tests such as state accountability measures, with a special eye for computer‐based simulation tasks.

On the Roles of External Knowledge Representations in Assessment Design

“…Recent work on "assessment engineering" (e.g., Luecht, 2002; aims not only to make the underlying principles explicit, but also to embed the underlying principles in design KRs that help assessment professionals structure, and at times automate, their work (Mislevy & Haertel, 2006). Assessment design KRs thus concern the how of assessment.…”

“…We then further develop and illustrate two of these roles, namely the design of assessment tasks around domain KRs and the creation of special KRs that help the assessment designer accomplish this. We place this discussion in the context of evidence-centered assessment design (ECD; Mislevy, Steinberg, & Almond, 2003;Mislevy & Haertel, 2006) to take advantage of KRs emerging from that work.…”

Application of Evidence-Centered Assessment Design to the Advanced Placement Redesign: A Graphic Restatement

“…The figure correlates the process with the stages of assessment design in evidence centered design (ECD), as they were outlined in Mislevy and Haertel (2006). The figure also introduces two validity terms not referred to by the presenters, construct representation and nomothetic span (Embretson, 1983), that I will attempt to embed in the process used with the Advanced Placement ® (AP ® ) Exams.…”

Research on Validity Theory and Practice at ETS

“…ECD starts with an analysis of the attributes, or constructs, of interest and the social and cognitive contexts in which they function and then designs the assessment to generate the kinds and amounts of evidence needed to draw the intended inferences. The ECD framework involves several stages of analysis (Mislevy and Haertel 2006;Mislevy et al 1999Mislevy et al , 2002Mislevy et al , 2003a. The first stage, domain analysis, concentrates on building substantive understanding of the performance domain of interest, including theoretical conceptions and empirical research on student learning and performance, and the kinds of situations in which the performances are likely to occur.…”