An investigation into the optimal number of distractors in single-best answer exams

Abstract

In UK medical schools, five-option single-best answer (SBA) questions are the most widely accepted format of summative knowledge assessment. However, writing SBA questions with four effective incorrect options is difficult and time consuming, and consequently, many SBAs contain a high frequency of implausible distractors. Previous research has suggested that fewer than five-options could hence be used for assessment, without deterioration in quality. Despite an existing body of empirical research in this area however, evidence from undergraduate medical education is sparse. The study investigated the frequency of non-functioning distractors in a sample of 480 summative SBA questions at Cardiff University. Distractor functionality was analysed, and then various question models were tested to investigate the impact of reducing the number of distractors per question on examination difficulty, reliability, discrimination and pass rates. A survey questionnaire was additionally administered to 108 students (33 % response rate) to gain insight into their perceptions of these models. The simulation of various exam models revealed that, for four and three-option SBA models, pass rates, reliability, and mean item discrimination remained relatively constant. The average percentage mark however consistently increased by 1–3 % with the four and three-option models, respectively. The questionnaire survey revealed that the student body had mixed views towards the proposed format change. This study is one of the first to comprehensively investigate distractor performance in SBA examinations in undergraduate medical education. It provides evidence to suggest that using three-option SBA questions would maximise efficiency whilst maintaining, or possibly improving, psychometric quality, through allowing a greater number of questions per exam paper.

Five-Option vs Four-Option Multiple-Choice Questions

“…This result is not in line with the finding by Hingorjo & Jaleel (2012) that 42% of 50 MCQs with five options were free from NFDs and the finding by Tarrant et al (2009) that 12.3% of 514 MCQs questions with four options were free from NFDs. These findings are neither in line with the finding by Kilgour & Tayyaba (2016), who analyzed 480 questions with 1920 distracters and revealed that 34 questions (7.1%) performed with 4 FDs.…”

“…For them, test items should consist of as many options as is feasible as regards the item content and the number of plausible distracters. Kilgour & Tayyaba (2016) explored non-functioning distractors in a sample of 480 fiveoption SBA (single best answer) questions of 327 university students. The study followed a two-step procedure, where the first step was to determine the frequency of non-functioning distractors across the sample of exam paper by analyzing the frequency of selection at the below 5 % level.…”

Analysis of MCQ and distractor use in a large first year Health Faculty Foundation Program: assessing the effects of changing from five to four options

“…Furthermore, many four- or five-option MCQs suffer from having ineffective distractors, that is, answers that are so implausible that these answers are rarely chosen [ 2 , 7 , 11 ]. Prior studies in nursing and medical programs have examined the effect of reducing MCQ options by modelling effects and redistribution of marks [ 4 , 11 ]. Other studies examined sequential cohorts: Tarrant and Ware examined a single undergraduate public health nursing course (142 examinees), with some re-used questions [ 10 ].…”

Comparison between three option, four option and five option multiple choice question tests for quality parameters: A randomized study

“…[ 6 7 8 ] Constructing good MCQ item is a complex, challenging, and time consuming process particularly for finding plausible distracters. [ 8 9 ] MCQs offer another disadvantage that chances of guessing increases with the decrease in number of options. [ 8 ] There are various types of MCQs used in the assessment in education-single best option type/multiple option, extended matching items, patient vignettes, etc., Single best response MCQs can also be divided further based on number of options such as three responses, four responses, and five responses.…”