Focus Article: On the Structure of Educational Assessments

Mislevy, Robert J.; Steinberg, Lynne; Almond, Russell G.

doi:10.1207/s15366359mea0101_02

Cited by 504 publications

(366 citation statements)

References 21 publications

Supporting

Mentioning

356

Contrasting

Unclassified

Order By: Relevance

“…Since the early 1990s, researchers have developed several related approaches to validation (Kane 1992(Kane , 2006(Kane , 2013aMislevy 2006Mislevy , 2009Mislevy et al 1999;Mislevy et al 2003b;Shepard 1993) that have sought to streamline models of validity and to add some more explicit guidelines for validation by stating the intended interpretation and use of the scores in the form of an argument. The argument would provide an explicit statement of the claims inherent in the proposed interpretation and use of the scores (Cronbach 1988).…”

Section: Argument-based Approaches To Validationmentioning

confidence: 99%

Research on Validity Theory and Practice at ETS

Kane

Bridgeman

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

Educational Testing Service (ETS) was founded with a dual mission: to provide high-quality testing programs that would enhance educational decisions and to improve the theory and practice of testing in education through research and development (Bennett 2005; Educational Testing Service 1992). Since its inception in 1947, ETS has consistently evaluated its testing programs to help ensure that they meet high standards of technical and operational quality, and where new theory and new methods were called for, ETS researchers made major contributions to the conceptual frameworks and methodology.This chapter reviews ETS's contributions to validity theory and practice at various levels of generality, including overarching frameworks (Messick 1988(Messick , 1989, more targeted models for issues such as fairness, and particular analytic methodologies (e.g., reliability, equating, differential item functioning). The emphasis will be on contributions to the theory of validity and, secondarily, on the practice of validation rather than on specific methodologies. Validity TheoryGeneral conceptions of validity grew out of basic concerns about the accuracy of score meanings and the appropriateness of score uses (Kelley 1927), and they have necessarily evolved over time as test score uses have expanded, as proposed interpretations have been extended and refined, and as the methodology of testing has become more sophisticated.

show abstract

Section: Argument-based Approaches To Validationmentioning

confidence: 99%

Research on Validity Theory and Practice at ETS

Kane

Bridgeman

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

show abstract

“…The increasingly ambitious assessments that were being conceived in the 1990s stimulated a rethinking of the assessment design process and led to the conception of ECD (Mislevy et al 2003). In addition, ETS expanded its research agenda to include the role of assessment in instruction and forms of assessment that, in a sense, are beyond format.…”

Section: Summary and Reflectionsmentioning

confidence: 99%

A Historical Survey of Research Regarding Constructed-Response Formats

Bejar

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

This chapter chronicles ETS research and development contributions related to the use of constructed-response item formats. 1 The use of constructed responses in testing dates back to imperial China, where tests were used in the selection of civil servants. However, in the United States, the multiple-choice format became dominant during the twentieth century, following its invention and use by the SAT ® examinations created by the College Board in 1926. When ETS was created in 1947, post-secondary admissions testing was largely based on tests consisting of multiplechoice items. However, from the start, there were two camps at ETS: those who believed that multiple-choice tests were sufficiently adequate for the purpose of assessing "verbal" skills and those who believed that "direct" forms of assessment requiring written responses had a role to play. For constructed-response formats to regain a foothold in American education several hurdles would need to be overcome. Research at ETS was instrumental in overcoming those hurdles.The first hurdle was that of reliability, specifically the perennial issue of low interrater agreement, which plagued the acceptance of constructed-response formats for most of the twentieth century. The second hurdle was broadening the conception of validity to encompass more than predictive considerations, a process that began with the introduction of construct validity by Cronbach and Meehl (1955). Samuel Messick at ETS played a crucial role in this process by making construct validity relevant to educational tests. An inflexion point in the process of reincorporating constructed-response formats more widely in educational tests was marked 1 Constructed responses to a prompt or question can range in scope and complexity. Perhaps the most common constructed response is the written essay. However, short written responses to questions are also considered to be constructed, as are spoken answers in response to a prompt, mathematical responses (equations, plotted functions, etc.), computer programs, and graphical responses such as architectural designs.

show abstract

“…Evidence-centered assessment design (ECD; Mislevy, Steinberg, & Almond, 2002, 2003) was developed in part to provide assessment designers with ways of talking about the new kinds of concerns raised by new kinds of assessments. To wean assessment designers away from assumptions that are not necessarily true in new types of assessments, ECD deliberately introduced new language.…”

Section: Extending Evidence-centered Assessment Design To Embedded Asmentioning

confidence: 99%

“…ECD, as described in Mislevy et al (2003), is simultaneously a process for designing assessments, a set of principles for addressing the design decisions that arise, and a formal language for describing the design. The latter is called the conceptual assessment framework (CAF, Figure 3), and this section will briefly review the principle design objects of the CAF, called models.…”

Section: A Brief Review Of Evidence-centered Assessment Designmentioning

confidence: 99%

How Task Features Impact Evidence From Assessments Embedded in Simulations and Games

Almond¹,

Kim²,

Velasquez³

et al. 2014

Measurement: Interdisciplinary Research and Perspectives

Self Cite

View full text Add to dashboard Cite

One of the key ideas of evidence-centered assessment design (ECD) is that task features can be deliberately manipulated to change the psychometric properties of items. ECD identifies a number of roles that task-feature variables can play, including determining the focus of evidence, guiding form creation, determining item difficulty and discrimination, characterizing proficiency, and producing task variants. Assessment developers can use these task features to manipulate the psychometric properties of both conventional assessment formats and complex tasks embedded in simulations and games.Simulations and games tasks present additional challenges: even defining what corresponds to an item can be difficult. Often task features are determined by game or simulation logic rather than psychometric design. Despite these difficulties, the roles for task features identified in ECD are useful for analyzing the psychometric properties of embedded assessments in simulations and games. This article compares a conventional test of mathematics-problem-solving ability using word problems to an assessment of conceptual physics, creativity, and conscientiousness embedded in the game Newton’s Playground, describing how the task roles play out in each setting and how they can be used to manipulate the evidence provided by an assessment.

show abstract

Focus Article: On the Structure of Educational Assessments

Cited by 504 publications

References 21 publications

Research on Validity Theory and Practice at ETS

Research on Validity Theory and Practice at ETS

A Historical Survey of Research Regarding Constructed-Response Formats

How Task Features Impact Evidence From Assessments Embedded in Simulations and Games

Contact Info

Product

Resources

About