Factors underlying performance on written tests of knowledge

Abstract

The study compares two popular forms of written tests; the multiple choice test (MCQ) and the Modified Essay Question (MEQ). Two factors were varied in the experiment: the format of the questions (multiple choice, directed free response, or open-ended free response) and the context of the questions (in a patient problem or in random sequence). Six problems were developed in each version, and administered to a total of 36 medical students at three educational levels using a Latin-square design. The results showed a significant effect of each factor in the design, amounting to a difference of 8.7% between MCQ and directed free response, 4.2% between directed and open-ended free response and 4.3% between problem and random context. However, the correlation of scores based on content across the formats approached unity after correction for attenuation. A process score, based on the style and presentation in the undirected format, correlated more strongly with the free-response questions. The results suggest that, although the MCQ and MEQ may assess different skills, there is a very strong relationship between content scores derived from the two formats. The free response formats may present the opportunity for assessment of other factors related to presentation if scoring procedures are modified. Finally, the effect of randomizing questions is a deterioration of performance when compared to placing questions in the problem context.

The Relationship Between Experience With Standardized Patient Examinations and Subsequent Standardized Patient Examination Performance

“…The concept of test-wiseness and its impact on exam validity has been studied extensively in the education literature (Geiger, 1997). In the medical education literature, the effect of test-taking skill and test-taking strategy on the results of other test formats such as multiple choice exams has also been extensively studied (Burton, 2002;Fischer, Herrmann, & Kopp, 2005;Harden, Brown, Biran, Ross, & Wakeford, 1976;Norman et al, 1987;Schuwirth & Van der Vlueten, 2004). In one example, it was demonstrated that the majority of changes (55%) to answers on a standardized multiple choice test of medical knowledge were from correct to incorrect and that changing answers is a superior testing strategy compared to trusting the first answer that comes to mind (Fischer et al, 2005).…”

Effects of Quantity of Instruction on Time Spent on Learning and Achievement

“…However, from a perspective from ecological validity, it seems clear that this format is generally used and accepted at the Maastricht Medical School, many other medical schools or institutions for higher education. Previous studies in medical education (Norman et al, 1987;Stalenhoef et al, 1990) have shown that a high correlation exists between examinee scores in open-ended and objective questions (multiple-choice or true-false format). Different formats of written tests provided similar information about students.…”

L'évaluation du raisonnement clinique

“…Il est en effet nécessaire, en raison de la spécificité de contenu, de mettre l'accent sur les phases de raisonnement réellement cruciales, de façon à gagner du temps d'examen et de pouvoir multiplier les cas évalués 7,8,32 . Les données de la recherche montrent qu'il est essentiel de se préoccuper de la tâche cognitive que doit effectuer l'étudiant, beaucoup plus que du format (39) qui recueille la réponse de l'étudiant (QCM, réponse ouve rte et courte, choix dans une longue liste d'options, ou TCS). Cette tâche doit être suffisamment complexe pour requérir un réel processus de résolution de problème.…”