Pick-N multiple choice-exams: a comparison of scoring algorithms

Abstract

To compare different scoring algorithms for Pick-N multiple correct answer multiple-choice (MC) exams regarding test reliability, student performance, total item discrimination and item difficulty. Data from six 3rd year medical students' end of term exams in internal medicine from 2005 to 2008 at Munich University were analysed (1,255 students, 180 Pick-N items in total). Scoring Algorithms: Each question scored a maximum of one point. We compared: (a) Dichotomous scoring (DS): One point if all true and no wrong answers were chosen. (b) Partial credit algorithm 1 (PS(50)): One point for 100% true answers; 0.5 points for 50% or more true answers; zero points for less than 50% true answers. No point deduction for wrong choices. (c) Partial credit algorithm 2 (PS(1/m)): A fraction of one point depending on the total number of true answers was given for each correct answer identified. No point deduction for wrong choices. Application of partial crediting resulted in psychometric results superior to dichotomous scoring (DS). Algorithms examined resulted in similar psychometric data with PS(50) only slightly exceeding PS(1/m) in higher coefficients of reliability. The Pick-N MC format and its scoring using the PS(50) and PS(1/m) algorithms are suited for undergraduate medical examinations. Partial knowledge should be awarded in Pick-N MC exams.

Rating Scale Measures in Multiple-Choice Exams: Pilot Studies in Pharmacology

“…ey rather recommend the Pick "N" format, in which the examinees are instructed to select "N" responses [3]. In addition, it has been argued that Pick-N format questions might be better than type-A questions because in clinical practice there is often more than one acceptable solution to a given problem [18,19]. We did, on purpose, not make the examinees aware how many responses they were supposed to pick, in order to make guessing a less useful strategy.…”

“…We did, on purpose, not make the examinees aware how many responses they were supposed to pick, in order to make guessing a less useful strategy. Others have studied different scoring scales or scoring algorithms for Pick-N-type MCQ [18]. Firstly, dichotomous scoring: one point if all true and no wrong answers were chosen.…”

“…irdly, a fraction (1/m) of one point was given for each correct answer given. ey concluded that the second and third options had higher statistical reliability and thus recommended to award partial knowledge in these type (Pick-N) 8 Education Research International of exams (medical students in Munich, internal medicine tested, Germany [18]). Others came to similar conclusions by showing that the third scoring method also attained higher validity, but the first method exhibited higher reliability (medical students in the USA [19]).…”

“…Hence, fewer questions are needed to obtain reliability, thus saving time for faculty. ese Pick-N-type questions were, however, also different from our format in that more than five answers were available [3,18]. e number of options to be selected is explicitly indicated in Pick-N-type, because otherwise scoring problems can occur.…”

Evaluation of student assessment practices in a medical college

“…Although it is known that MCQs allow for test reliability and validity, a major disadvantage of them is that they test for recall of facts rather than application if not well constructed. 12,13 However, developing well written MCQs is not easy and could be very challenging to staff and needs regular training. 14 In our context, this might need more measures to be taken to motivate the staff.…”