A Brief Introduction to Evidence‐centered Design

Abstract

Evidence-centered assessment design (ECD) is an approach to constructing educational assessments in terms of evidentiary arguments. This paper provides an introduction to the basic ideas of ECD, as well as some of the terminology and models that have been developed to implement the approach. In particular, it presents the high-level models of the Conceptual Assessment Framework (CAF) and the Four-process Delivery Architecture for assessment delivery systems. Special attention is given to the role of probability-based reasoning in accumulating evidence across task performances, in terms of belief about unobservable variables that characterize the knowledge, skills, and/or abilities of students. This is the role traditionally associated with psychometric models, such as those of item response theory (IRT) and latent class models. To unify the ideas and to provide a foundation for extending probability-based reasoning in assessment applications more broadly, however, a more general expression in terms of graphical models is indicated. This brief overview of evidence-centered design provides the reader with a feel for where and how graphical models fit into the larger enterprise of educational and psychological assessment. A simple example based on familiar large-scale standardized tests such as the Graduate Record Examinations ® (GRE ® ) is used to fix ideas.Key words: Assessment design, delivery system, evidence, psychometrics i iii OverviewWhat all educational assessments have in common is the desire to reason from particular things students say, do, or make, to broader inferences about their knowledge and abilities. Over the past century, a number of assessment methods have evolved for addressing this problem in a principled and systematic manner. The measurement models of classical test theory and, more recently, item response theory (IRT) and latent class analysis, have proved quite satisfactory for the large scale tests and classroom quizzes with which every reader is by now quite familiar.But off-the-shelf assessments and standardized tests are increasingly unsatisfactory for guiding learning and evaluating students' progress. Advances in cognitive and instructional sciences stretch our expectations about the kinds of knowledge and skills we want to develop in students and the kinds of observations we need to evidence them (Glaser, Lesgold, & Lajoie, 1987). Advances in technology make it possible to evoke evidence of knowledge more broadly conceived and to capture more complex performances. One of the most serious bottlenecks we face, however, is making sense of complex data that result.Fortunately, advances in evidentiary reasoning (Schum, 1994) and in statistical modeling (Gelman, Carlin, Stern, & Rubin, 1995) allow us to bring probability-based reasoning to bear on the problems of modeling and uncertainty that arise naturally in all assessments. These advances extend the principles upon which familiar test theory is grounded to more varied and complex inferences from more complex data . One canno...

How and when does complex reasoning occur? Empirically driven development of a learning progression focused on complex reasoning about biodiversity

“…Step Three: Learning Progression-Mapped Development of Assessment Items Our assessment design followed principles of Evidence Centered Design (ECD; Mislevy et al, 2004) towards the creation of pretests, embedded assessments, and posttests calibrated on the same scale. A major tenet of the ECD approach is the recognition that assessment is a vehicle for the gathering of empirical evidence so that we can make inferences about unobserved phenomena; in this case our content and inquiry reasoning progressions that we mentioned earlier cannot be evaluated directly.…”

Research on Validity Theory and Practice at ETS

“…Mislevy (1993Mislevy ( , 1994Mislevy ( , 1996Mislevy ( , 2007 focused on the role of evidence in validation, particularly in terms of model-based reasoning from observed performances to more general claims about students and other test takers. Mislevy et al (1999Mislevy et al ( , 2002Mislevy et al ( , 2003a developed an ECD framework that employs argument-based reasoning. ECD starts with an analysis of the attributes, or constructs, of interest and the social and cognitive contexts in which they function and then designs the assessment to generate the kinds and amounts of evidence needed to draw the intended inferences.…”

“…ECD starts with an analysis of the attributes, or constructs, of interest and the social and cognitive contexts in which they function and then designs the assessment to generate the kinds and amounts of evidence needed to draw the intended inferences. The ECD framework involves several stages of analysis (Mislevy and Haertel 2006;Mislevy et al 1999Mislevy et al , 2002Mislevy et al , 2003a. The first stage, domain analysis, concentrates on building substantive understanding of the performance domain of interest, including theoretical conceptions and empirical research on student learning and performance, and the kinds of situations in which the performances are likely to occur.…”

Multimedia-based performance assessment in Dutch vocational education

“…The big challenge lies in investigating the validity of simulations as measurement instruments. A very useful point of departure for research into the theory and practice of SBA is the evidence-centered design (ECD) framework (Mislevy, Almond, & Lukas, 2004). Therefore, before the challenges of SBA will be discussed in greater detail, the ECD framework, and in particular the conceptual assessment framework (CAF) within the ECD approach will be introduced (see Figure 1.1).…”

“…Another challenge for using SBA lies in defining and specifying coherent and complete psychometric models that fit the data that students' performance in SBA's produces. A useful point of departure for this discussion is the conceptual assessment framework (CAF) layer within the evidence-centered design framework (ECD) (Mislevy, Almond, & Lukas, 2004). The CAF consists of three separate, though strongly related, models: the student model, the activity model and the evidence model.…”