Self-monitoring and its relationship to medical knowledge

Abstract

In the domain of self-assessment, researchers have begun to draw distinctions between summative self-assessment activities (i.e., making an overall judgment of one's ability in a particular domain) and self-monitoring processes (i.e., an "in the moment" awareness of whether one has the necessary knowledge or skills to address a specific problem with which one is faced). Indeed, previous research has shown that, when responding to both short answer and multiple choice questions, individuals are able to assess the likelihood of answering questions correctly on a moment-by-moment basis, even though they are not able to generate an accurate self-assessment of overall performance on the test. These studies, however, were conducted in the context of low-stakes tests of general "trivia". The purpose of the present study was to further this line of research by investigating the relationship between self-monitoring and performance in the context of a high stakes test assessing medical knowledge. Using a recent administration of the Medical Council of Canada Qualifying Examination Part I, we examined three measures intended to capture self-monitoring: (1) the time taken to respond to each question, (2) the number of questions a candidate flagged as needing to be considered further, and (3) the likelihood of changing one's initial answer. Differences in these measures as a function of the accuracy of the candidate's response were treated as indices of each candidate's ability to judge his or her likelihood of responding correctly. The three self-monitoring indices were compared for candidates at three different levels of overall performance on the exam. Relative to correct responses, when examinees initially responded incorrectly, they spent more time answering the question, were more likely to flag the question for future consideration, and were more likely to change their answer before committing to a final answer. These measures of self-monitoring were modulated by candidate performance in that high performing examinees showed greater differences on these indices relative to poor performing examinees. Furthermore, reliability analyses suggest that these difference measures hold promise for reliably differentiating self-monitoring at the level of individuals, at least within a given content area. The results suggest that examinees were self-monitoring their knowledge and skills on a question by question basis and altering their behavior appropriately in the moment. High performing individuals showed stronger evidence of accurate self-monitoring than did low performing individuals and the reliability of these measures suggests that they have the potential to differentiate between individuals. How these findings relate to performance in actual clinical settings remains to be seen.

How well do doctors think they perform on the General Medical Council's Tests of Competence pilot examinations? A cross-sectional study

“…Candidates’ self-monitoring was measured according to time taken to respond to each question, the number of questions flagged for further consideration and the likelihood of changing their initial answer. This study found that high performers demonstrated better self-monitoring than poorer performers on the examination 17. Following this evidence, there are recommendations to pursue this line of research approach instead of asking people to estimate their own examination scores 14–17…”

“…Psychology students showed awareness of the limits of their knowledge by spending longer time on questions they were unsure about and avoiding answering questions they knew they would get incorrect 15. A study with medical students who took a qualifying examination reported similar findings 17. Candidates’ self-monitoring was measured according to time taken to respond to each question, the number of questions flagged for further consideration and the likelihood of changing their initial answer.…”

Exploring Clinical Reasoning Strategies and Test-Taking Behaviors During Clinical Vignette Style Multiple-Choice Examinations: A Mixed Methods Study

“…12 For admitting knowledge deficits, the increased likelihood of this behavior among high Step 2 CK performers is consistent with research demonstrating superior self-monitoring by high-performing medical student test-takers. 27 During analysis of the think-aloud transcripts, we found admitting knowledge deficits as an attitudinal difference among participants. Although not definitive from the results, it is plausible that premature closure and failure to admit knowledge deficits, and their increased likelihood with low Step 2 CK performance, may relate to overconfidence.…”

Study Strategies Are Associated with Performance in Basic Science Courses in the Medical Curriculum

“…The study skills and strategies of students (including medical students) have been the subject of numerous investigations (Amin, Tani, Eng, Samarasekara, & Huak, 2009;Barker & Olson, 1997;Karpicke, Butler, & Roediger, 2009;McConnell, Regehr, Wood, & Eva, 2011;Newble & Gordon, 1985;Pandey & Zimitat, 2007;Reid, Duvall, & Evans, 2005;Salamonson, Everett, Koch, Wilson, & Davidson, 2009;P. Weinstein & Gipple, 1974;Wenger, Hobbs, Williams, Hays, & Ducatman, 2009;West & Sadoski, 2011), but there is little evidence of associations between study strategies and performance in medical courses (P. Weinstein & Gipple, 1974;West & Sadoski, 2011).…”