Comparison of long-menu and single-best-answer multiple choice questions in computer-based summative assessments: a randomised controlled trial

Abstract

Background Little is known regarding the psychometric properties of computerized long-menu formats in comparison to classic formats. We compared single-best-answer (Type A) and long-menu formats using identical question stems during the computer-based, summative, intermediate clinical-clerkship exams for nine disciplines. Methods In this randomised sequential trial, we assigned the examinees for every summative exam to either the Type A or long-menu format (four different experimental questions, otherwise identical). The primary outcome was the power of discrimination. The study was carried out at the Faculty of Medicine, University of Geneva, Switzerland, and included all the students enrolled for the exams that were part of the study. Examinees were surveyed about the long-menu format at the end of the trial. Results The trial was stopped for futility ( p = 0.7948) after 22 exams including 88 experimental items. The long-menu format had a similar discriminatory power but was more difficult than the Type A format (71.45% vs 77.80%; p = 0.0001). Over half of the options (54.4%) chosen by the examinees in long-menu formats were not proposed as distractors in the Type A formats. Most examinees agreed that their reasoning strategy was different. Conclusions In a non-selected population of examinees taking summative exams, long-menu questions have the same discriminatory power as classic Type A questions, but they are slightly more difficult. They are perceived to be closer to real practice, which could have a positive educational impact. We would recommend their use in the final years of the curriculum, within realistic key-feature problems, to assess clinical reasoning and patient management skills.

Implementation of Computer-Based Summative Assessment

“…The needs identification stage involves analyzing the issues with paper-based assessment that are less effective and efficient. In this phase, the school management team successfully identified these problems and studied the available school resources and assets (Bello et al, 2021;Cerutti et al, 2019;Mitra et al, 2015). In the planning phase, the school management team formulated specific objectives, identified targets and participants, and designed strategies and activities to be carried out.…”

Online assessment in higher education: a mapping review and narrative synthesis

“…Studies at the augmentation level (n=60) employed online tests to gain better understanding of learners' current knowledge and provide additional learning opportunities. There was an emphasis on using online technologies to enable timely and often automated feedback (Herbert et al, 2019) or design tests that incorporated multimedia resources or sophisticated types of questions (Cerutti et al, 2019). These studies unanimously reported student appreciation of formative feedback from online tests (Chen & Chuang, 2012).…”

Notfälle in der Augenheilkunde: Vermittlung anhand interaktiver Key-feature-Fälle für Medizinstudierende

“…Überraschend für uns war, dass zwar in der Evaluation die Informationsmenge, welche zur Verfügung gestellt wurde, für die Lösung als genau richtig bewertet wurde, in den Freitextkommentaren jedoch darüber hinausgehend deutlich mehr Hin-Abb. 3…”

“…Die Erstellung der Fälle gelang hierbei mit moderatem technischem Aufwand und geringen Investitionskosten. Wünschenswert wären allerdings weitere Frageformate gewesen, beispielsweise eine Long-Menu-Option hätte die Quizabschnitte deutlich aufgewertet [ 3 ]. Da jedoch aufgrund der verpflichtend digitalen Lehre im Rahmen der Corona-Pandemie keine Zeit für eine separate Programmierung und Implementierung entsprechender Features war, musste dies für die 1.…”