Scaling, Linking, and Reporting in a Periodic Assessment System

Abstract

A new entry in the testing lexicon is through‐course summative assessment, a system consisting of components administered periodically during the academic year. As defined in the Race to the Top program, these assessments are intended to yield a yearly summative score for accountability purposes. They must provide for both individual and group proficiency estimates and allow for the measurement of growth. They must accommodate students who vary in their patterns of curricular exposure. Because they are meant to provide actionable information to teachers they must be instructionally sensitive, so item‐operating characteristics can be expected to change relative to one another as a function of patterns of curricular exposure. This paper discusses methodology one can draw upon to tackle this ambitious collection of inferences. We consider a modeling framework that consists of an item response theory component and a population component, as in the National Assessment of Educational Progress, and show how performance and growth could be expressed in terms of expected performance on a market basket of tasks. We discuss conditions under which modeling simplifications might be possible and discuss studies that would be needed to fit models, estimate parameters, and evaluate data requirements.

The Changing Nature of Educational Assessment

“…The assessments would be designed to be engaging and to be valuable learning experiences in and of themselves. Because there would be many of them, they would be administered at various times throughout the school year, with the results aggregated within-student for purposes of representing achievement (much as teachers aggregate evidence to award an individual’s course grade, or the major leagues aggregate wins and losses to determine which team goes to the championships; see Mislevy & Zwick, 2012, for more on the significant psychometric issues attendant to such aggregation). Also due to their number, each assessment would have considerably less influence, in contrast to the onetime test of today, eliminating the problem of a student or class having a “bad day.” These new assessments might also incidentally provide tentative formative results, pointing teachers toward a student’s placement in a learning progression and how that student’s problem-solving processes might be improved, results to be followed up with more targeted, teacher-directed data gathering.…”

A Sociocognitive Perspective on Assessing EL Students in the Age of Common Core and Next Generation Science Standards

“…Measuring growth is particularly problematic because higher or lower performance depends on the particular language uses, topic, process, and cross‐cutting theme experiences a student has had before and between time points . Multidimensional measurement models can help sort out the differing profiles across students (Jang, ; Mislevy & Zwick, ), but even so, the tasks provide far less information about individual students than when contextualization and conditional inference can be employed. This is a design tradeoff between broadly comparable evidence and individually useful evidence.…”

Modeling Growth With Adaptive Longitudinal Large‐Scale Assessments

“…von Davier, Xu, and Carstensen () used a general latent variable model to measure growth for the Programme for International Student Assessment (PISA). Mislevy and Zwick () discussed the scaling and linking issues in reporting the results of a periodic assessment system. Some growth models assessed growth by the changes of item parameters (Doran & Jiang, ; Kaplan & Sweetman, ; Reckase, ; von Davier, ), for example, the Embretson model (Embretson, ; Xu & Qian, ).…”