A Brief Introduction to Evidence-Centered Design

Digital information literacy (DIL)—generally defined as the ability to obtain, understand, evaluate, and use information in a variety of digital technology contexts—is a critically important skill deemed necessary for success in higher education as well as in the global networked economy. To determine whether college graduates possess the requisite knowledge and skills in DIL, higher education institutions must be able to administer and use results from valid assessments of DIL. In this paper, we provide a comprehensive review of existing definitions of this construct in major frameworks from higher education and the workforce and propose an operational definition of DIL. Next, we provide a review of existing assessments of information literacy and related constructs, including features of the assessments, construct alignment, and psychometric properties (i.e., reliability and validity evidence). Finally, we discuss challenges and considerations surrounding the design, implementation, and use of next‐generation assessments of DIL. We offer this review as a resource for higher education institutions in selecting among existing assessments or in designing their own measures.

Section: Review Of Existing Assessments Of Digital Information Literacymentioning

confidence: 99%

Assessing Digital Information Literacy in Higher Education: A Review of Existing Frameworks and Assessments With Recommendations for Next‐Generation Assessment

Sparks

Katz

Beile

2016

“…SBAs form part of a continuum of innovative assessments that include complex computerized simulations, games, and other complex performance tasks; build on cognitive analysis of task requirements; and consider a variety of sources of evidence, including data from log files and other process measures (Agostinho, Meek, & Herrington, ; Gobert, Sao Pedro, Baker, Toto, & Montalvo, ; Herrington & Oliver, ; Herrington, Oliver, & Reeves, ; Mislevy, ; Mislevy et al, ; Mislevy et al, ; Rupp et al, ; Rupp et al, ; Shute et al, ; Zane, , ). They differ from more open‐ended assessment systems such as games and simulations by adopting a more traditional, linear sequence of items—which is, however, mapped onto a meaningful task structure derived from an analysis of the key practice.…”

Section: The Link Between Key Practices and Scenario‐based Assessmentmentioning

confidence: 99%

“…The process by which the CBAL ELA framework and existing CBAL ELA assessments were developed falls solidly within the tradition of evidence‐centered design (ECD; Mislevy, Almond, & Lukas, ; Mislevy & Haertel, ), which emerged in large part from efforts to extend the theory of assessment design to cover nontraditional assessments, including simulations and other performance assessments (see, e.g., Frezzo, Behrens, & Mislevy, ; Mislevy, ; Mislevy et al, ; Mislevy, Steinberg, Breyer, Almond, & Johnson, ; Rupp, Gushta, Mislevy, & Shaffer, ; Shute, Masduki, & Donmez, ; West et al, ; Zalles, Haertel, & Mislevy, ).…”

mentioning

confidence: 99%

Key Practices in the English Language Arts (ELA): Linking Learning Theory, Assessment, and Instruction

Deane

Sabatini

Feng

et al. 2015

This paper presents a framework intended to link the following assessment development concepts into a systematic framework: evidence‐centered design (ECD), scenario‐based assessment (SBA), and assessment of, for, and as learning. The context within which we develop this framework is the English language arts (ELA) for K‐12 students, though the framework could easily be applied to cover reading, writing, and critical thinking skills from pre‐K through college. Central to the framework is the concept of a key practice, drawn from constructivist learning theory, which emphasizes the purposeful social context within which skills are recruited and organized to carry out complex literacy tasks. We argue that key practices provide a key link between existing CBAL™ ELA learning progressions (defined as part of a student model for literacy skills) and the structure of well‐designed SBAs. This structure enables us to design assessments that model a key practice, supporting the systematic creation of task sequences that can be used to support both instruction and assessment.

“…The development team for the Elementary Education: CKT Science component of the ETS Educator Series used evidence‐centered design (ECD) principles to develop the test content specifications for this new licensure assessment (Mislevy & Riconscente, ). ECD has been described as a “methodology for designing assessments that underscores the central role of evidentiary reasoning in assessment design” (Mislevy, Almond, & Lukas, , p. 20). In particular, ECD emphasizes the importance of test specifications as a direct outgrowth of considerations regarding the test's purpose, the test‐taking population, the use of the scores, what knowledge is sufficient for safe and effective entry into the profession, the assessment items needed to elicit evidence about test takers' knowledge, and how this evidence will be used to support claims that test takers possess sufficient knowledge for entry‐level practice (Tannenbaum et al, ).…”

Section: Methods and Data Sourcesmentioning

confidence: 99%

Assessing Elementary Teachers' Content Knowledge for Teaching Science for theETS® Educator Series: Pilot Results

Mikeska

Kurzum

Steinberg

et al. 2018

The purpose of this report is to examine the performance of assessment items designed to measure elementary teachers' content knowledge for teaching (CKT) science as part of the ETS® Educator Series. The Elementary Education: CKT Science assessment is 1 component of licensure examination through the PRAXIS® assessments. The Elementary Education: CKT Science assessment is designed to determine whether kindergarten through 6th‐grade elementary teacher candidates have the essential content knowledge needed for teaching elementary science as new teachers at the entry level. This report provides information about the development of the Elementary Education: CKT Science framework and the associated assessment items. The main part of the report focuses on the evidence gathered when piloting 104 CKT science assessment items with 417 preservice and novice elementary teachers. This evidence included the following: (a) how these new CKT science assessment items function, including their item difficulties and discrimination, and reliabilities of the pilot forms and of the classification of examinees; (b) how teachers perceive the importance and relevance of these new assessment items; (c) how teachers' performances on these items relate to their background characteristics and professional and academic preparation; and (d) how teachers' performances on the CKT science items compare to their performances on an assessment designed to measure only their science subject matter knowledge.