Self-assessment: With all its limitations, why are we still measuring and teaching it? Lessons from a scoping review

Abstract

Self-assessment: with all its limitations, why are we still measuring and teaching it? Lessons from a scoping review. Intro: Self-Assessment (SA) is often assumed to be essential for learning, however this assumption has been extensively rebutted. Research shows SA has significant limitations, including its lack of correlation with competence. Method:We undertook a scoping review of SA in medical education (2011)(2012)(2013)(2014)(2015)(2016)(2017)(2018)(2019)(2020)(2021) and surprisingly discovered substantial research where SA was erroneously assumed to be a valid measure of successful learning, or a skill needing to be taught. Although the initial intent of our scoping review was to explore where SA is being effectively used to advance lifelong learning, we paused to explore the extent of the problem of its misuse, and resulting wasted research, by examining excluded studies.Results: From 1151 articles, we identified 207 which ignored the documented limitations of SA. 39 studies explored SA of learning. This research has limited utility: increasing accuracy of SA does not improve performance or lifelong learning. 168 used SA as an outcome measure to assess a program or intervention, including 63 where self-assessed improvement in knowledge/skills was the sole measure. SA of self-confidence was measured in 62 studies. When confidence was compared with an objective measure of performance, both invariably increased, but confidence did not always align with competence when measured.Discussion: Many researchers mistakenly assume the ability to accurately selfassess is essential for learning, so focus on teaching self-calibration rather than evidence-based methods of advancing learning. Other researchers incorrectly suppose that self-reported improvements in knowledge/skills provides evidence of efficacy of a program/intervention. This is particularly troubling with regards to novices, who may believe that because they self-assess to have improved/gained confidence, they are now competent.

Behavioral skills training for teaching safety skills to mental health service providers compared to training-as-usual: a pragmatic randomized control trial

“…Our finding of a differential impact of training on confidence versus demonstrable skills is consistent with a long-standing, substantial body of research examining the relationship between self-assessment and objective measures of learning [ 28 , 65 , 66 ]. The pattern of non-existent, weak, or even inverse relationships between the two has been shown for a variety of medical staff trainee and education learner groups [ 28 , 29 , 67 – 72 ].…”

Flexible bronchoscopic intubation through a supraglottic airway device: An evaluation of consultant anaesthetist performance

“…Studies have demonstrated that selfassessment of knowledge and skill performance is often inadequate and lack correlation with competence. 19 In medical education, there is a dearth of evidence on how mastery of airway skills is attained, and life-long learning achieved. 20 We found a positive correlation between previous experience with FBI and performance, but volume of practice does not necessarily translate into level of expertise.…”

“…In our study, physicians received no feedback between cases. Studies have demonstrated that self‐assessment of knowledge and skill performance is often inadequate and lack correlation with competence 19 …”

Informed self-assessment during the transition to medical school: a longitudinal narrative study

“…This is because self-assessment accuracy of a skill does not correlate with improved performance in that skill. 16 Instead, attention has moved to understanding the process of ISA as a strategy for learning, and how this can be supported. Changing patterns in self-assessment describe, but do not explain how and why medical students learn to balance, synthesise and reconcile available data to inform self-assessment.…”

“…The focus in self-assessment research has shifted away from studies of accuracy. This is because self-assessment accuracy of a skill does not correlate with improved performance in that skill 16. Instead, attention has moved to understanding the process of ISA as a strategy for learning, and how this can be supported.…”