The Role of Instructional Sensitivity in the Empirical Review of Criterion‐referenced Test Items

Haladyna, Tom; Roid, Gale H.

doi:10.1111/j.1745-3984.1981.tb00841.x

Cited by 36 publications

(34 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…To examine instructional sensitivity of the instrument, we applied the paired samples t ‐test at α = 0.05 to Rasch ability estimates of students who took the temporal‐magnitude instrument before and after the cosmic evolution course. Instructional sensitivity in this study is defined as “the tendency for an item to vary in difficulty as a function of instruction” (Haladyna & Roid, 1981, p. 40). If the instrument was sensitive to the cosmic evolution course that targeted scientific changes with extremely long durations, students would improve their temporal magnitude recognition ability after the cosmic evolution course by mainly improving their performances on the extremely long duration items.…”

Section: Methodssupporting

confidence: 88%

College students' temporal‐magnitude recognition ability associated with durations of scientific changes

et al. 2010

View full text Add to dashboard Cite

The purpose of this study was to explore college students' recognition of temporal magnitudes associated with durations of scientific changes through construct validation of a 30‐item instrument. We administered the instrument to 514 students from 10 higher education institutions in the United States. Among them, 419 students took the instrument at the end of science courses. The remaining 95 students took the instrument before and after a course on cosmic evolution, and additionally answered whether they knew, estimated, or randomly guessed at the temporal magnitude of each item in the instrument. We also collected exam scores for the cosmic evolution course students. Using descriptive statistics and a Rasch analysis, we investigated construct validity of the instrument in terms of psychological relevance, psychometric conformity, and instructional sensitivity. Results of this study indicate that (1) the temporal‐magnitude recognition ability is a measurable construct, (2) extremely small duration items are significantly more difficult for students to recognize accurate temporal magnitudes than other duration range items, (3) direct knowledge of the magnitude contributes to the measurement of the construct, (4) the instrument is sensitive to instruction designed to improve the construct, and (5) the temporal‐magnitude recognition ability is not significantly correlated with knowledge about the related scientific changes. © 2010 Wiley Periodicals, Inc., Inc. J Res Sci Teach 48: 317–335, 2011

show abstract

Section: Methodssupporting

confidence: 88%

College students' temporal‐magnitude recognition ability associated with durations of scientific changes

et al. 2010

View full text Add to dashboard Cite

show abstract

“…Polikoff's (2010) review of instructional-sensitivity research notes pervasive differences among items' operating characteristics before and after instruction (e.g., see Muthén, Kao, & Burstein, 1991;Tatsuoka, Linn, Tatsuoka, & Yamamoto, 1988;Wilson, 1989). He also reaffirms Haladyna and Roid's (1981) finding that "sensitivity indices were not systematically related to the difficulty or discrimination indices" (Polikoff, 2010, p. 7), implying the need for a model more complex than one that is adequate for a single time point. Lord (1976) describes the relationship between instructional sensitivity and dimensionality as follows:…”

Section: Considerations For the Irt Modelmentioning

confidence: 97%

Scaling, Linking, and Reporting in a Periodic Assessment System

Mislevy

Zwick

2012

J Educational Measurement

View full text Add to dashboard Cite

A new entry in the testing lexicon is through‐course summative assessment, a system consisting of components administered periodically during the academic year. As defined in the Race to the Top program, these assessments are intended to yield a yearly summative score for accountability purposes. They must provide for both individual and group proficiency estimates and allow for the measurement of growth. They must accommodate students who vary in their patterns of curricular exposure. Because they are meant to provide actionable information to teachers they must be instructionally sensitive, so item‐operating characteristics can be expected to change relative to one another as a function of patterns of curricular exposure. This paper discusses methodology one can draw upon to tackle this ambitious collection of inferences. We consider a modeling framework that consists of an item response theory component and a population component, as in the National Assessment of Educational Progress, and show how performance and growth could be expressed in terms of expected performance on a market basket of tasks. We discuss conditions under which modeling simplifications might be possible and discuss studies that would be needed to fit models, estimate parameters, and evaluate data requirements.

show abstract

“…Most methods required item difficulty estimates before and after students had received instruction, or difficulty estimates for equivalent groups of students who had or had not received instruction. Several empirical methods to compare pre-and post-item difficulties or item difficulties for the two groups have been proposed, including a simple item p value comparison procedure (Cox & Vargas, 1966), a phi-coefficient analysis (Popham, 1971), a t test of item response theory (IRT) item calibration differences (Wright & Stone, 1979), and Bayesian methods (see Haladyna & Roid, 1981, for a review of those methods).…”

Section: Background and Objectives The Evolving Concept Of Instructiomentioning

confidence: 99%

Instructional Sensitivity of a State's Standards-Based Assessment

D’Agostino

Welsh

Corson

2007

Educational Assessment

View full text Add to dashboard Cite

The accuracy of achievement test score inferences largely depends on the sensitivity of scores to instruction focused on tested objectives. Sensitivity requirements are particularly challenging for standards-based assessments because a variety of plausible instructional differences across classrooms must be detected. For this study, we developed a new method for capturing the alignment between how teachers bring standards to life in their classrooms and how the standards are defined on a test. Teachers were asked to report the degree to which they emphasized the state's academic standards, and to describe how they taught certain objectives from the standards. Two curriculum experts judged the alignment between how teachers brought the objectives to life in their classrooms and how the objectives were operationalized on the state test. Emphasis alone did not account for achievement differences among classrooms. The best predictors of classroom achievement were the match between how the standards were taught and tested, and the interaction between emphasis and match, indicating that test scores were sensitive to instruction of the standards, but in a narrow sense.

show abstract

The Role of Instructional Sensitivity in the Empirical Review of Criterion‐referenced Test Items

Cited by 36 publications

References 18 publications

College students' temporal‐magnitude recognition ability associated with durations of scientific changes

College students' temporal‐magnitude recognition ability associated with durations of scientific changes

Scaling, Linking, and Reporting in a Periodic Assessment System

Instructional Sensitivity of a State's Standards-Based Assessment

Contact Info

Product

Resources

About