Quality Multiple-Choice Test Questions: Item-Writing Guidelines and an Analysis of Auditing Testbanks

Hansen, James D.; Lee, Dexter

doi:10.1080/08832329709601623

Cited by 46 publications

(49 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Tarrant et al (2006) evaluated 2,770 MCQs used over a five-year period from 2001 to 2005 and concluded that nearly half (46%) of the items were bad because they violated item-writing guidelines. Similar outcomes in higher education assessments are reported across different disciplines (Ellsworth et al, 1990;Hansen and Dexter, 1997;Masters et al, 2001;Downing, 2005). Poorly written MCQs can negatively impact students' performance and achievement (Downing, 2005;Tarrant et al, 2006;Clifton and Schriner, 2010).…”

Section: Quality Of Mcqs In Higher Educationsupporting

confidence: 49%

Evaluating the Quality of Higher Education Instructor-Constructed Multiple-Choice Tests: Impact on Student Grades

Brown

Abdulnabi

2017

Front. Educ.

View full text Add to dashboard Cite

Multiple-choice questions (MCQs) are commonly used in higher education assessment tasks because they can be easily and accurately scored, while giving good coverage of instructional content in a short time. However, studies that have evaluated the quality of MCQs used in higher education assessments have found many flawed items, resulting in misleading insights about student performance and contaminating important decisions. Thus, MCQs need to be evaluated statistically to ensure high-quality items are used as the basis of inference. This study evaluated the quality of 100 instructor-written MCQs used in an undergraduate midterm test (50 items) and final exam (50 items), making up 50% of the course grade, using the responses of 372 students enrolled in one first-year undergraduate general education course. Item difficulty, discrimination, and chance properties were determined using classical test theory and item response theory (IRT) statistical item analysis models. The two-parameter logistic (2PL) model consistently had the best fit to the data. The impact on overall course grades between the original raw score model and the IRT 2PL model showed 70% of students would receive the same grade (i.e., D to A), but only one-third would get the same mark using the standard augmented grade scale (i.e., A+ to D−). The analyses show that higher education institutions need to ensure MCQs are evaluated before student grading decisions are made.

show abstract

Section: Quality Of Mcqs In Higher Educationsupporting

confidence: 49%

Evaluating the Quality of Higher Education Instructor-Constructed Multiple-Choice Tests: Impact on Student Grades

Brown

Abdulnabi

2017

Front. Educ.

View full text Add to dashboard Cite

show abstract

“…Likewise, we believe that instructors have a responsibility to avail themselves of this training in order to ensure that their MC tests are well-constructed and have acceptable discriminatory power (Whitley, Perkins, Balogh, Keith-Spiegel, & Wittig, 2000). The creation of high-quality MC items is a learnable skill (Hansen, 1997;Jozefowicz et al, 2002), and by working together, institutions and instructors can improve the state of MC testing at Canadian universities.…”

Section: Discussionmentioning

confidence: 99%

“…Deviating from the guidelines can be problematic because it can detract from the quality of individual items and of the test as a whole (Downing, 2005;Tarrant & Ware, 2008). As it happens, failure to conform to the guidelines is widespread in the MC items found both on tests in postsecondary classrooms (Jozefowicz et al, 2002;Tarrant, Knierim, Hayes, & Ware, 2006) and in publisher-supplied test banks (Hansen & Dexter, 1997;Masters et al, 2001).…”

mentioning

confidence: 99%

Examination of the Quality of Multiple-choice Items on Classroom Tests

DiBattista

Kurzawa

2011

cjsotl-rcacea

View full text Add to dashboard Cite

Because multiple-choice testing is so widespread in higher education, we assessed the quality of items used on classroom tests by carrying out a statistical item analysis. We examined undergraduates’ responses to 1198 multiple-choice items on sixteen classroom tests in various disciplines. The mean item discrimination coefficient was +0.25, with more than 30% of items having unsatisfactory coefficients less than +0.20. Of the 3819 distractors, 45% were flawed either because less than 5% of examinees selected them or because their selection was positively rather than negatively correlated with test scores. In three tests, more than 40% of the items had an unsatisfactory discrimination coefficient, and in six tests, more than half of the distractors were flawed. Discriminatory power suffered dramatically when the selection of one or more distractors was positively correlated with test scores, but it was only minimally affected by the presence of distractors that were selected by less than 5% of examinees. Our findings indicate that there is considerable room for improvement in the quality of many multiple-choice tests. We suggest that instructors consider improving the quality of their multiple-choice tests by conducting an item analysis and by modifying distractors that impair the discriminatory power of items.Étant donné que les examens à choix multiple sont tellement généralisés dans l’enseignement supérieur, nous avons effectué une analyse statistique des items utilisés dans les examens en classe afin d’en évaluer la qualité. Nous avons analysé les réponses des étudiants de premier cycle à 1198 questions à choix multiples dans 16 examens effectués en classe dans diverses disciplines. Le coefficient moyen de discrimination de l’item était +0.25. Plus de 30 % des items avaient des coefficients insatisfaisants inférieurs à + 0.20. Sur les 3819 distracteurs, 45 % étaient imparfaits parce que moins de 5 % des étudiants les ont choisis ou à cause d’une corrélation négative plutôt que positive avec les résultats des examens. Dans trois examens, le coefficient de discrimination de plus de 40 % des items était insatisfaisant et dans six examens, plus de la moitié des distracteurs était imparfaits. Le pouvoir de discrimination était considérablement affecté en cas de corrélation positive entre un distracteur ou plus et les résultatsde l’examen, mais la présence de distracteurs choisis par moins de 5 % des étudiants avait une influence minime sur ce pouvoir. Nos résultats indiquent que les examens à choix multiple peuvent être considérablement améliorés. Nous suggérons que les enseignants procèdent à une analyse des items et modifient les distracteurs qui compromettent le pouvoir de discrimination des items

show abstract

“…Open banks of exam questions are published in numerous countries and every student can become familiar with their contents and thus prepare for the final exam or mid-term test (Considine et al, 2005;Hansen & Dexter, 1997;Park & Yang, 2015;Yang et al, 2018). It needs to be noted, however, that open task banks usually contain a few thousand of items (e.g.…”

Section: Discussionmentioning

confidence: 99%

Does Repetition of the Same Test Questions in Consecutive Years Affect their Psychometric Indicators? – Five-year Analysis of In-house Exams at Medical University of Warsaw

Pańczyk

Zarzeka

Malczyk

et al. 2018

EURASIA J MATH SCI T

View full text Add to dashboard Cite

Aim of study: Evaluation of the re-used test questions on the impact of psychometric indicators of test items in examinations in cardiology in Physiotherapy at the Medical University of Warsaw (MUW). Materials and Methods:A case study based on analysis of 132 five-option (528 distractors) multiple-choice questions (MCQs) developed at MUW included in five inhouse exams. Questions repeated at least twice during the period considered and constituted 42.4% of all MCQs. Each MCQ was assessed on the basis of the following three indicators: difficulty index (DI), discrimination power (DP), and the number of non-functioning distractors (N-FD). The change in psychometric indicators of test items was assessed using Krippendorff alpha coefficient (αk). Results:Together with each MCQs repetition, a decrease in the number of questions that would maintaining the analogical DI value towards the initial level of easiness was observed. However, the level of DI compliance was significantly higher, even when there were five consecutive repetitions (coefficient αk for the consecutive repetitions was 0.90, 0.85, 0.78 and 0.75). N-FD number in consecutive repetitions remained on a satisfactory level (good and very good compliance), although there was a significant decrease in this range when there were three or more repetitions (coefficient αk was 0. 80, 0.69, 0.66 and 0.65, respectively). Whereas the level of similarity as for DP for consecutive repetitions was significantly lower in comparison with those noted for DI and DE (DP coefficient αk was 0.28, 0.23, 0.25 and 0.10, respectively). Conclusions:The observed change in the initial values of psychometric indicators together with consecutive use of the same MCQs confirms the examiners' concerns as for the progressive wear of the bank of test questions. However, the level of psychometric MCQs values loss, especially in the area of the easiness and the number of non-functioning distractors was not drastic. It appears that the level of MCQs spread among students of consecutive years is not too high, at least within two consecutive years.

show abstract

Quality Multiple-Choice Test Questions: Item-Writing Guidelines and an Analysis of Auditing Testbanks

Cited by 46 publications

References 4 publications

Evaluating the Quality of Higher Education Instructor-Constructed Multiple-Choice Tests: Impact on Student Grades

Evaluating the Quality of Higher Education Instructor-Constructed Multiple-Choice Tests: Impact on Student Grades

Examination of the Quality of Multiple-choice Items on Classroom Tests

Does Repetition of the Same Test Questions in Consecutive Years Affect their Psychometric Indicators? – Five-year Analysis of In-house Exams at Medical University of Warsaw

Contact Info

Product

Resources

About