Understanding self-assessment as an informed process: residents’ use of external information for self-assessment of performance in simulated resuscitations

Abstract

Self-directed learning requires self-assessment of learning needs and performance, a complex process that requires collecting and interpreting data from various sources. Learners' approaches to self-assessment likely vary depending on the learner and the context. The aim of this study was to gain insight into how learners process external information and apply their interpretation of this information to their self-assessment and learning during a structured educational activity. The study combined quantitative performance data with qualitative interview data. Pediatric residents led video-recorded simulated resuscitations and rated their crisis resource management skills on a validated 6-item instrument. Three independent observers rated the videos using the same instrument. During semi-structured interviews, each resident reviewed the video, rerated performance, discussed the self-assessment process, and interpreted feedback and observer scores. Transcripts were analyzed for themes. Sixteen residents participated. Residents' self-assessed scores ranged widely but usually fell within two points of the observers. They almost universally lowered their scores when self-assessing after the video review. Five major themes emerged from qualitative analysis of their interviews: (1) residents found self-assessment important and useful in certain contexts and conditions; (2) residents varied in their self-directed learning behaviors after the simulated resuscitation; (3) quantitative observer assessment had limited usefulness; (4) video review was difficult but useful; and (5) residents focused on their weaknesses and felt a need for constructive feedback to enhance learning. The residents in our study almost uniformly embraced the importance of self-assessment for all medical professionals. Even though video review had a negative impact on their self-assessment scores and was perceived as painful, residents saw this as the most useful aspect of the study exercises residents. They were less accepting of the quantitative assessment by observers. Residents explained their tendency to focus on weaknesses as a way to create an incentive for learning, demonstrating that self-assessment is closely linked to self-directed learning. How learners can use video review and external assessment most effectively to guide their self-directed learning deserves further study.

Learner-Driven Synthesis of Assessment Data: Engaging and Motivating Residents in Their Milestone-Based Assessments

“…However, there is potential concern about whether self-assessment using milestones will help to inform resident progress or to facilitate self-directed learning skills. Research over the past decade has shown that selfassessment may encourage some residents to improve their awareness of deficiencies, create individualized learning plans, and bolster specific knowledge areas [4][5][6][7][8][9]. Unfortunately, self-assessment is fraught with inaccuracy and varies depending on the setting, content, and learner perspective [10][11][12].…”

“…To our knowledge, this is not a common practice in the milestone-based assessment era. Prior studies looking at this type of informed self-assessment (providing residents with external feedback for the sake of synthesizing assessments) have found that it minimizes the flaws inherent in primary self-assessment reliability while capturing the value of learner investment in the assessment process [5,7,23].…”

Self-assessment and dialogue: can it improve learning?

“…Although it is known from previous studies that self-assessment requires clear, timely, specific and constructive feedback to inform the learners' self-assessment (Sargeant et al 2010); not much is known about the conditions under which self-assessment is effective and does enhance learning. Plant et al (2012) report an interesting study about the process of self-assessment. The aim of the study was to better understand HOW and WHY resident physicians adjust their self-assessment after reviewing their own performance in leading a simulated resuscitation in the presence of an interviewer.…”

“…So, whether self-assessment can improve learning cannot be answered based on the study of Plant et al (2012) as the authors also mention in their discussion. Nevertheless, this study demonstrates that qualitative feedback is crucial but also that dialogue, either in a group or with a personal coach, stimulates reflections.…”

Authors reply to commentary by Dolmans

“…

Authors replyThe authors would like to thank Dr. Dolmans for her thoughtful commentary on our study examining residents' use of external information during self-assessment of their performance (Plant et al 2012).In her piece entitled ''Self-assessment and dialogue: can it improve learning? '', Dr. Dolmans (2012) discusses the importance of dialogue in the self-assessment process.

…”