Nonfunctioning Options: A Closer Look

Abstract

In a previous study, Cizek and O'Day reported the effects of eliminating nonfunctioning options from five-option, multiple-choice test items. That study, however, only investigated the effect of option removal on item performance when items were studied as a set and not the effects of option removal for individual items. The authors also did not examine potential changes in dimensionality resulting from altering the items. The present study provides a closer look at the effect of removing nonfunctioning options by examining changes in difficulty, discrimination, and dimensionality. Results of these analyses supported the conclusions of the earlier research and provide additional support for benefits that may accrue from eliminating nonfunctioning options: enhanced score reliability, reduced testing time, potential for broader domain sampling, and consequent increased validity of score interpretation.

Fixed or mixed: a comparison of three, four and mixed-option multiple-choice tests in a Fetal Surveillance Education Program

“…This information has since been fed back into item writing workshops, where inclusion of ‘opt out’ options to make up option numbers has been discouraged. Generally, the results presented in this article are consistent with those reported by Cizek, Robinson and O’Day [14] where test-level variation is modest but item-level variation can be considerable, following a reduction of one response option.…”

“…Instead, there was appreciable variation at the individual item level. These findings mirror those in another medical education assessment context [14], and contribute to the idea that these trends are generalisable. This discouraged the recommendation of a blanket policy for the number of options.…”

“…For a smaller number (11) of new four-option items that had yet to be trialled, subject-matter experts eliminated what they perceived to be the least plausible incorrect option, based on experience with similarly styled/structured items, to produce a three-option item. Cizek and O’Day [14] showed that subject-matter experts’ selections can be highly consistent with empirical data about these relative frequencies. In compiling the first two test forms, exactly the same items in terms of the item stem appeared in each test.…”

“…When the number of items is fixed and the number of options is manipulated, mixed results and recommendations are reported; while some studies have identified a small or negligible change in item difficulty or discrimination for different option numbers [7,14-17], others have found decreases in difficulty and discrimination for smaller option numbers [3]. The results depend on the quality of the removed response options.…”

Using of Structural Equation Modeling Techniques in Cognitive Levels Validation

“…It is likely that the very high discriminations were the common factor or the variable that created high covariations among the latent variables. This situation is not uncommon in the test preparation and there are numerous examples where the psychometric characteristics of items, such as difficulty or discrimination, strongly affect the dimensionality of a test [30][31].…”