Validity of very short answer versus single best answer questions for undergraduate assessment

Abstract

BackgroundSingle Best Answer (SBA) questions are widely used in undergraduate and postgraduate medical examinations. Selection of the correct answer in SBA questions may be subject to cueing and therefore might not test the student’s knowledge. In contrast to this artificial construct, doctors are ultimately required to perform in a real-life setting that does not offer a list of choices. This professional competence can be tested using Short Answer Questions (SAQs), where the student writes the correct answer without prompting from the question. However, SAQs cannot easily be machine marked and are therefore not feasible as an instrument for testing a representative sample of the curriculum for a large number of candidates. We hypothesised that a novel assessment instrument consisting of very short answer (VSA) questions is a superior test of knowledge than assessment by SBA.MethodsWe conducted a prospective pilot study on one cohort of 266 medical students sitting a formative examination. All students were assessed by both a novel assessment instrument consisting of VSAs and by SBA questions. Both instruments tested the same knowledge base. Using the filter function of Microsoft Excel, the range of answers provided for each VSA question was reviewed and correct answers accepted in less than two minutes. Examination results were compared between the two methods of assessment.ResultsStudents scored more highly in all fifteen SBA questions than in the VSA question format, despite both examinations requiring the same knowledge base.ConclusionsValid assessment of undergraduate and postgraduate knowledge can be improved by the use of VSA questions. Such an approach will test nascent physician ability rather than ability to pass exams.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-016-0793-z) contains supplementary material, which is available to authorized users.

Evaluation of a Flipped Examination Model Implemented in a Final-Year Undergraduate Pharmacotherapeutics Course

“…Additionally, while MCQs are widely used in undergraduate and postgraduate health care medical examinations, the addition of short-answer questions could possibly enrich the learning experience and test the students' nascent abilities rather than their ability to pass exams. 37 Since data on the design, format, and potential impact of flipped exams in pharmacy education is generally scarce, further research is recommended before conclusions could be generalized. The popularity of flipped classrooms seems to be increasing in higher education, and more promising results are expected in the future, particularly in terms of refining knowledge retention, encouraging personal accountability, and improving student learning.…”

Very Short Answer Questions: A Viable Alternative to Multiple Choice Questions

“…Multiple True/False (MTF) and One Best Answer Questions (BAQ) are widely employed in medical faculties by virtue of their advantages of instant machine scoring, freedom from examiner bias and dependable reliability [1][2][3][4]. In this article, 'MCQ' is used to refer to both these instruments of assessment.…”

“…Reliability of a test is higher when the subject coverage is wider [5,6]. Reliability refers to test reproducibility with similar results when used for different batches, but it does not ensure validity [2,3]. MCQ tests seem to sacrifice validity for reliability [3].…”

“…MCQ is blamed for promoting rote learning, guessing, test-wiseness, and turning students into data banks [2,5,7,[12][13][14]. MCQ tend to test trivia [2] and they are not able to test complex issues [3,7]. MCQ does not provide any opportunity for students to express their understanding [15].…”

The use of distractors in multiple-choice questions: a medical student perspective

“…Poor-quality distractors in MCQs could allow students to arrive at the answer by a process of elimination as opposed to deduction. Furthermore, students may rely on cues and implausible distractors,3 circumventing the need for thorough understanding. It should also be noted that Rahma et al and Sam et al both recognized that creating four good distractors may be challenging for examiners, especially when core knowledge is being tested 1,3.…”