Modified Multiple-Choice Items for Alternate Assessments: Reliability, Difficulty, and Differential Boost

Abstract

Federal policy on alternate assessment based on modified academic achievement standards (AA-MAS) inspired this research. Specifically, an experimental study was conducted to determine whether tests composed of modified items would have the same level of reliability as tests composed of original items, and whether these modified items helped reduce the performance gap between AA-MAS eligible and ineligible students. Three groups of eighth-grade students (N = 755) defined by eligibility and disability status took original and modified versions of reading and mathematics tests. In a third condition, the students were provided limited reading support along with the modified items. Changes in reliability across groups and conditions for both the reading and mathematics tests were determined to be minimal.

Considering the Use of General and Modified Assessment Items in Computerized Adaptive Testing

“…was piloted in the Consortium for Alternate Assessment and Experimental Studies (CAAVES) project and found to help enhance access and yield more meaningful scores for students that take AA-MAS (Elliott et al, , 2013Kettler, Elliott, & Beddow, 2009;Kettler et al, 2011;. Many other states have created guidelines similar to TAMI to simplify and improve the quality of their AA-MAS (see Bowman, 2013;Texas Education Agency, 2009).…”

Effects of Reducing the Cognitive Load of Mathematics Test Items on Student Performance

“…A recent series of studies tested the effects of modified items on the performance of students with severe disabilities (Kettler et al 2009(Kettler et al , 2011. These researchers modified items using CLT, Universal Design guidelines (Rose and Meyer 2000) and research on item development.…”

Distractor Functioning in Modified Items for Test Accessibility

“…The study included a carefully balanced design with rotation of modified items from the beginning to the end of the test across students, such that each student was exposed to items in original and modified format. The results of this study have been analyzed in terms of shifts in item difficulty and test score reliability (Elliott et al, 2010;Kettler et al, 2011). Although there were significant group by condition interaction effects on test score reliability, none of the differences (less than .06) were very meaningful.…”

“…Many item stems were edited; had graphics modified, removed, or added; and were reformatted in a number of ways to improve accessibility, as described above. More information on this process can be found in Kettler et al (2011). Changes to item and distractor statistics must be interpreted with this process in mind.…”

“…Using the Rasch model to control for individual (and group) ability, the changes in average item difficulty were evaluated for each group (see Kettler et al, 2011, for a complete review of results). Generally, the Rasch model produces a scale score with a standard deviation of 1.0 and negative change in item difficulty indicates the items become easier (requiring less ability to respond correctly).…”