Comparison of Difficulties and Reliabilities of Quantitative Word Problems in Completion and Multiple-Choice Item Formats

Oosterhof, Albert; Coats, Pamela K.

doi:10.1177/014662168400800305

Cited by 21 publications

(15 citation statements)

References 10 publications

(8 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We found that NOTA significantly increased difficulty when it replaced the key. This finding is consistent with previous studies that have demonstrated an increase in difficulty with inclusion of NOTA (Crehan & Haladyna, 1991;Dudycha & Carpenter, 1973;Oosterhof & Coats, 1984;Tollefson, 1987). However, unlike all but a select few studies (Dudycha & Carpenter, 1973;Rich & Johanson, 1990), we have demonstrated a decrease in discrimination coefficients following NOTA inclusion.…”

Section: Discussionsupporting

confidence: 92%

“…Most studies have found that NOTA increases test difficulty (Crehan & Haladyna, 1991;Dudycha & Carpenter, 1973;Forsyth & Spratt, 1980;Knowles & Welch, 1992;Oosterhof and Coats, 1984;Rimland, 1960;Tollefson, 1987;Wesman & Bennett, 1946). However, it is important to distinguish between the presence of NOTA as the key or as a distractor.…”

Section: Introductionmentioning

confidence: 99%

“…In some studies demonstrating an effect of NOTA on item difficulty, the difference between NOTA as a key and NOTA as a distractor was not considered, and as a result these studies demonstrate relatively small effect sizes (Rimland, 1960). In other studies, NOTA increased difficulty when it was the key, but not when it was a distractor (Oosterhof & Coats, 1984;Tollefson, 1987). Another common measure of multiple choice item efficacy is the discrimination coefficient, which assesses how well a particular item distinguishes between high performing and low performing students as measured by their overall score on the test.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Systematic Assessment of ‘None of the Above’ on Multiple Choice Tests in a First Year Psychology Classroom

2015

View full text Add to dashboard Cite

Multiple choice writing guidelines are decidedly split on the use of ‘none of the above’ (NOTA), with some authors discouraging and others advocating its use. Moreover, empirical studies of NOTA have produced mixed results. Generally, these studies have utilized NOTA as either the correct response or a distractor and assessed its effect on difficulty and discrimination. In these studies, NOTA commonly yields increased difficulty when it is used as the correct response, and no change in discrimination regardless of usage. However, when NOTA is implemented as a distractor, rarely is consideration given to the distractor that could have been written in its place. Here, we systematically replaced each distractor in a series of questions with NOTA across different versions of an Introductory Psychology examination. This approach allowed us to quantify the quality of each distractor based on its relative discrimination index and assess the effect of NOTA relative to the quality of distractor it replaced. Moreover, our use of large Introductory Psychology examinations afforded highly stable difficulty and discrimination estimates. We found that NOTA increased question difficulty only when it was the correct response, with no effect on difficulty of replacing any distractor type with NOTA. Moreover, we found that NOTA decreased discrimination when it replaced the most effective distractors, with no effect on discrimination of replacing either the correct response or lowest quality distractor with NOTA. These results replicate the common finding that inclusion of NOTA as the correct response increases question difficulty by equally luring high-performing and low-performing students toward distractors. Moreover, we have shown that including NOTA as a distractor can reduce discrimination if used in lieu of a well written alternative, suggesting that multiple choice authors should avoid using NOTA on multiple choice tests. Les guides de rédaction pour les questions à choix multiple sont incontestablement partagés sur l’usage de la réponse « aucune des situations ci-dessus ». Certains auteurs déconseillent de l’employer alors que d’autres préconisent de le faire. De plus, des études empiriques de l’emploi de cette expression ont mené à des résultats mitigés. En général, ces études ont utilisé l’option « aucune des situations ci-dessus » soit comme étant la réponse correcte soit comme distracteur et ont évalué ses effets sur la difficulté et le discernement. Dans ces études, la réponse « aucune des situations ci-dessus » mène généralement à une augmentation de la difficulté quand l’option est employée comme étant la réponse correcte et il n’y a aucun changement en ce qui concerne le discernement, quel que soit son usage. Toutefois, quand ce type d’option de réponse est utilisé en tant que distracteur, on trouve rarement une justification à l’emploi d’un distracteur qui aurait pu être utilisé à sa place. Dans le cas présent, nous avons systématiquement remplacé chaque distracteur dans une série de questions contenant l’option « aucune des situations ci-dessus » dans différentes versions d’un examen d’introduction à la psychologie. Cette approche nous a permis de quantifier la qualité de chaque distracteur sur la base de l’indice de son discernement relatif et d’évaluer les effets de l’option relative sur la qualité du distracteur qu’elle remplaçait. De plus, puisque nous avons utilisé de grands examens d’introduction à la psychologie, cela nous a permis de faire des estimations de la difficulté et du discernement hautement stables. Nous avons trouvé que l’emploi de l’option « aucune des situations ci-dessus » augmentait la difficulté de la question seulement lorsque cette option était la réponse correcte et qu’il n’avait aucun effet sur la difficulté présente lorsqu’on remplaçait n’importe quel type de distracteur par l’option « aucune des situations ci-dessus ». En outre, nous avons trouvé que l’emploi de l’option « aucune des situations ci-dessus » diminuait le discernement quand elle remplaçait les distracteurs les plus efficaces et qu’elle n’avait aucun effet sur le discernement quand elle remplaçait soit la réponse correcte soit le distracteur le moins plausible par l’option « aucune des situations ci-dessus ». Ces résultats reproduisent les conclusions communes selon lesquelles l’emploi de l’option « aucune des situations ci-dessus » comme étant la réponse correcte augmente la difficulté de la question car dans ce cas, tant les étudiants brillants que les étudiants médiocres sont leurrés de façon identique vers les distracteurs. Par surcroît, nous avons montré que le fait d’inclure l’option « aucune des situations ci-dessus » en tant que distracteur pouvait réduire le discernement si on l’utilisait à la place d’une alternative de réponse bien rédigée, ce qui suggère que les auteurs de questions à choix multiples devraient éviter d’utiliser l’option « aucune des situations ci-dessus » dans les examens à choix multiples.

show abstract

Section: Discussionsupporting

confidence: 92%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Systematic Assessment of ‘None of the Above’ on Multiple Choice Tests in a First Year Psychology Classroom

2015

View full text Add to dashboard Cite

show abstract

“…Differences in item discrimination were reported by Forsyth and Sprat (1980), Oosterhof and Coats (1984), and Tollefson and Tripp (1983). But, Crehan and Haladyna (1991), Crehan, Haladyna, and Brewer (1993), Dudycha and Carpenter (1973), Frary (1991) and Tollefson (1987) did not find differences in item discrimination.…”

Section: Previous Research On Nota Itemsmentioning

confidence: 71%

“…These investigations were at some point contradictionary. Some reported differences in difficulty when using the NOTA option in comparison with a conventional option (Crehan & Haladyna, 1991;Crehan, Haladyna, & Brewer, 1993;Dudycha & Carpenter, 1973;Forsyth & Spratt, 1980;Frary, 1991;Oosterhof & Coats, 1984;Tollefson, 1987). Sometimes the differences were statistically significant, but not always.…”

Section: Previous Research On Nota Itemsmentioning

confidence: 99%

The assessment of quantitative problem-solving skills with “none of the above”-items (NOTA items)

et al. 2001

View full text Add to dashboard Cite

In this contribution we concentrate on the features of a particular item format: items having as the last option "none of the above" (NOTA items). There is considerable dispute on the advisability of the usage of NOTA items in testing. Some authors come to the conclusion that NOTA items should be avoided, some come to neutral conclusions while others argue that NOTA items are optimal test items. In this article, we provide evidence to this discussion by conducting protocol analysis on written statements of examinees while answering NOTA items. In our investigation, a test containing 30 multiple-choice items was administered from 169 university students. The results show that NOTA options appear to be more attractive than options with specified solutions in those cases where a problemsolver fails. Also, a relationship is found between the quality of (incorrect) problemsolving and the choice of NOTA items: the more qualitative the incorrect problemsolving process is, the more likely the student is to choose for NOTA items. Overall, our research supports the statement that 'the more confidence an examinee has in his worked solution, which is inconsistent with one of the specified solutions, the more eager he seems to choose "none ofthe above".

show abstract

Influence of True/False Tests and First Language on Engineering Students' Test Scores

Lackey

2002

J of Engineering Edu

View full text Add to dashboard Cite

True/false and open‐ended test scores were compared for an undergraduate engineering course. There was good correlation between the scores indicating that the use of true/false test items did not alter ranking of students regardless of whether or not their first language was English. Use of a mix of test item types is recommended since this permits testing over a greater fraction of the course content, encourages students to study, and is an efficient use of instructor time.

show abstract

Comparison of Difficulties and Reliabilities of Quantitative Word Problems in Completion and Multiple-Choice Item Formats

Cited by 21 publications

References 10 publications

A Systematic Assessment of ‘None of the Above’ on Multiple Choice Tests in a First Year Psychology Classroom

A Systematic Assessment of ‘None of the Above’ on Multiple Choice Tests in a First Year Psychology Classroom

The assessment of quantitative problem-solving skills with “none of the above”-items (NOTA items)

Influence of True/False Tests and First Language on Engineering Students' Test Scores

Contact Info

Product

Resources

About