Psychometric and Cognitive Analysis as a Basis for the Design and Revision of Quantitative Item Models

Abstract

We describe the item modeling development and evaluation process as applied to a quantitative assessment with high‐stakes outcomes. In addition to expediting the item‐creation process, a model‐based approach may reduce pretesting costs, if the difficulty and discrimination of model‐generated items may be predicted to a predefined level of accuracy. The development and evaluation of item models represents a collaborative effort among content specialists, statisticians, and cognitive scientists. A cycle for developing and revising item models that generate items with more predictable statistics is described. We review the goals of item modeling from different perspectives and recommend a method for structuring families of models that span content and generate items with more predictable psychometric parameters.

Untitled

“…One possibility is that items can be automatically generated by developing item models in which item features are purposefully manipulated so that specific aspects of the construct are tested and the item difficulties can be predicted. Some cognitive psychological research relates the psychometric properties, including item difficulties, of automatically generated items with their parent items, or the item models from which they are derived (Embretson & Daniel, 2008;Enright, Morley, & Sheehan, 2002;Graf, Peterson, Steffen, & Lawless, 2005;Newstead, Bradon, Handley, Dennis, & Evans, 2006). The concept of item generation would allow more attention to be paid, not only to the content in terms of test specifications, but also to the underlying construct itself.…”

Defining Mathematics Competency in the Service of Cognitively Based Assessment for Grades 6 Through 8

“…The general result from this work is that some models generate instances with very similar parameters while others generate instances with highly variable parameters, and how a model will behave is not always clear at the outset. Because of this, an iterative approach to item model development, where empirical evaluation is followed by subsequent revision, is recommended (Bejar, 1993;Bejar & Yocom, 1991;Embretson & Gorin, 2001;Graf, Peterson, Steffen, & Lawless, 2005).…”

“…Item modeling lends itself to being used in conjunction with experimental designs that systematically explore features that influence item difficulty and discrimination (Bejar, 1993;Bejar & Yocom, 1991;Embretson, 1999;Enright, Morley, & Sheehan, 2002;Graf et al, 2005;Newstead, Bradon, Handley, Evans, & Dennis, 2002).…”

Approaches to the Design of Diagnostic Item Models

“…The details for how to generalize a quantitative source item to an item model are described in Graf, Peterson, Steffen, and Lawless (2005). The parameterized source item then serves as a template for automatically generating instances (Bejar, 2002), or discrete items, from the item model.…”

“…At this point, quite a bit of research has systematically explored the impact of various item model variables on difficulty (e.g., Bejar, 1993;Bejar & Yocom, 1991;Embretson, 1999;Enright et al, 2002;Graf et al, 2005;Newstead, Bradon, Handley, Evans, & Dennis, 2002).…”