Difficulty and discrimination of multiple choice questions: A counterintuitive result

Sevenair, John P.; Burkett, Allan R.

doi:10.1021/ed065p441

Cited by 8 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a well established dome‐shaped relationship between the difficulty and discrimination indices of multiple‐choice questions (Sevenair and Burkett, 1988; Sim and Rasiah, 2006). Two of the questions used in this study were close to the plateau of the dome‐shaped relationship (Sim and Rasiah, 2006), with a discrimination index of 0.3–0.6 and a difficulty index of 0.5–0.7.…”

Section: Discussionmentioning

confidence: 99%

“…The high‐stakes examinations (proctored examinations that contributed to student course scores) were administered using a web‐based examination database and applications developed in‐house as part of LUMEN, the Loyola University Medical Education Network (McNulty et al, 2007) for technological details and examples of user interfaces. Reports for each examination included data on the difficulty and discrimination indices (Sevenair and Burkett, 1988; Sim and Rasiah, 2006) for each question. The difficulty factor ( D ) was calculated as D = c / n , where c = number of correct answers; n = number of respondents.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

An audience response system may influence student performance on anatomy examination questions

Hoyt

McNulty

Gruener

et al. 2010

Anatomical Sciences Ed

View full text Add to dashboard Cite

This study integrated an in-house audience response system (ARS) in the human anatomy course over two years to determine whether students performed better on high-stakes examinations following exposure to similar interactive questions in a large lecture format. Questions in an interactive ARS format were presented in lectures via PowerPoint presentations. Students who chose to participate in the anonymous ARS sessions submitted answers via their personal wireless devices (e.g., laptops, smartphones, PDAs, etc). Students were surveyed for feedback. Student participation in ARS activities was greatest (65-80%) in the first lecture. The number of students who actively participated in ARS activities decreased over the next four sessions, and then slightly increased in the last two sessions. This trend was the same for both years. Use of the ARS did not dramatically enhance overall student performance on examination questions that dealt with content similar to content presented in the ARS sessions. However, students who scored in the lower quartile of the examination performed better on the examination questions after the ARS was implemented. Accordingly, our findings suggest that the effect of ARS to improve student performance on examinations was not uniform. The overall benefit of an ARS to enhance the lecture experience was confirmed by student surveys.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

An audience response system may influence student performance on anatomy examination questions

Hoyt

McNulty

Gruener

et al. 2010

Anatomical Sciences Ed

View full text Add to dashboard Cite

show abstract

“…Difficulty was calculated as the number of correct responses divided by the total number of responses. Discrimination was calculated as the difference between the proportions of students in the top and bottom 27% of the class selecting the correct answer (Sevenair & Burkett, 1988). For those examination sets where there were two exam versions with dispersed questions (Exams 3, 4 and 5), the data from both dispersed versions were pooled prior to statistical evaluation.…”

Section: Methodsmentioning

confidence: 99%

“…Finally, the ability of an examination, as a whole, to accurately and reliably assess student knowledge and understanding can be determined by the calculation of the level of difficulty and discrimination associated with each MCQ composing that exam, a statistical evaluation that routinely accompanies computer-based exam grading (Lowe, 1991). The difficulty index reveals the percentage of students selecting the correct answer to each MCQ while the discrimination value, by comparing performance on each question between the higher-and lower-scoring students, rates the ability of each MCQ to be answered correctly more frequently by students who achieved higher overall scores on that particular exam (Lowe, 1991;Sevenair & Burkett, 1988).…”

Section: Introductionmentioning

confidence: 99%

Does Correct Answer Distribution Influence Student Choices When Writing Multiple Choice Examinations?

Carnegie

2017

cjsotl-rcacea

View full text Add to dashboard Cite

Summative evaluation for large classes of first- and second-year undergraduate courses often involves the use of multiple choice question (MCQ) exams in order to provide timely feedback. Several versions of those exams are often prepared via computer-based question scrambling in an effort to deter cheating. An important parameter to consider when preparing multiple exam versions is that they must be equivalent in their assessment of student knowledge. This project investigated a possible influence of correct answer organization on student answer selection when writing multiple versions of MCQ exams. The specific question asked was whether the existence of a series of four to five consecutive MCQs in which the same letter represented the correct answer had a detrimental influence on a student’s ability to continue to select the correct answer as he/she moved through that series. Student outcomes from such exams were compared with results from exams with identical questions but which did not contain such series. These findings were supplemented by student survey data in which students self-assessed the extent to which they paid attention to the distribution of correct answer choices when writing summative exams, both during their initial answer selection and when transferring their answer letters to the Scantron sheet for correction. Despite the fact that more than half of survey respondents indicated that they do make note of answer patterning during exams and that a series of four to five questions with the same letter for the correct answer would encourage many of them to take a second look at their answer choice, the results pertaining to student outcomes suggest that MCQ randomization, even when it does result in short serial arrays of letter-specific correct answers, does not constitute a distraction capable of adversely influencing student performance. Dans les très grandes classes de cours de première et deuxième années, l’évaluation sommative se déroule souvent par le biais d’examens comportant des questions à choix multiples afin de pouvoir donner rapidement les résultats aux étudiants. Plusieurs versions de ces examens sont souvent préparées et les questions sont brouillées par ordinateur pour dissuader la tricherie. Lors de la préparation de plusieurs versions d’un examen à choix multiples, l’un des paramètres importants à prendre en considération est que chaque version doit être semblable aux autres pour évaluer équitablement les connaissances des étudiants. Ce projet a pour but d’examiner l’influence possible de l’organisation des réponses correctes sur le choix des réponses des étudiants lors de la préparation de plusieurs versions d’un examen à choix multiples. La question spécifique qui a été posée était de savoir si l’existence d’une série de quatre ou cinq questions à choix multiples consécutives pour lesquelles la même lettre représentait la bonne réponse pouvait avoir une influence préjudiciable sur l’aptitude des étudiants à continuer à choisir la bonne réponse alors qu’ils progressent d’une question à l’autre dans la même série. Les résultats des étudiants qui passent de tels examens ont été comparés aux résultats obtenus quand les étudiants passent des examens dont les questions sont les mêmes mais qui ne comportent pas de telles séries. Ces résultats ont été enrichis par les réponses à une enquête auprès des étudiants pour laquelle les étudiants ont été auto-évalués concernant la question de savoir s’ils avaient remarqué la répartition des réponses correctes parmi les choix multiples quand ils passaient des examens sommatifs, à la fois au départ, quand ils choisissaient leurs réponses, et ensuite quand ils transféraient les lettres correspondant à leurs réponses sur la feuille Scanton pour la correction. Malgré le fait que plus de la moitié des répondants aient indiqué qu’ils ne font pas attention à la structuration des réponses pendant l’examen et qu’une série de quatre ou cinq questions ayant la même lettre pour la bonne réponse pourrait encourager beaucoup d’entre eux à regarder de plus près leur choix de réponse, la conclusion concernant les résultats obtenus par les étudiants suggère que la randomisation des questions à choix multiples, même quand elle aboutit à des séries de réponses correctes identifiées par la même lettre, ne constitue pas une distraction capable d’influencer négativement le rendement des étudiants.

show abstract

“…Both indices are useful in helping to define the reliability and validity of the examination items while also helping to construct an examination, based on prior psychometrics, to provide an estimated level of performance. These fields are automatically calculated each time the question is used based on embedded formulae (Sevenair & Burkett, 1988;Sim & Rasiah, 2006). Additionally, the indices provide immediate assistance to administrators in identifying problematic questions.…”

Section: Item Database and Examination Creation Applicationsmentioning

confidence: 99%

Computer-Based Testing in the Medical Curriculum: A Decade of Experiences at One School

McNulty

Chandrasekhar

Hoyt

et al. 2011

Journal of Educational Computing Research

View full text Add to dashboard Cite

This report summarizes more than a decade of experiences with implementing computer-based testing across a 4-year medical curriculum. Practical considerations are given to the fields incorporated within an item database and their use in the creation and analysis of examinations, security issues in the delivery and integrity of examinations, curricular integration, use of examination log files, effectiveness and timeliness of reports, and the potential for educational research. The solutions to many complicated issues have practical relevance and interest to the community of educators who are considering the adoption of this technology for assessment.

show abstract

Difficulty and discrimination of multiple choice questions: A counterintuitive result

Abstract: A report on a statistical study of how guessing affects test scores found that their results are contrary to expectations: average students are actually poorer at 'guessing' than students at the bottom of the class.

Cited by 8 publications

References 0 publications

An audience response system may influence student performance on anatomy examination questions

An audience response system may influence student performance on anatomy examination questions

Does Correct Answer Distribution Influence Student Choices When Writing Multiple Choice Examinations?

Computer-Based Testing in the Medical Curriculum: A Decade of Experiences at One School

Contact Info

Product

Resources

About