Milestone Implementation’s Impact on Narrative Comments and Perception of Feedback for Internal Medicine Residents: a Mixed Methods Study

Abstract

BACKGROUND: Feedback is a critical element of graduate medical education. Narrative comments on evaluation forms are a source of feedback for residents. As a shared mental model for performance, milestone-based evaluations may impact narrative comments and resident perception of feedback. OBJECTIVE: To determine if milestone-based evaluations impacted the quality of faculty members' narrative comments on evaluations and, as an extension, residents' perception of feedback. DESIGN: Concurrent mixed methods study, including qualitative analysis of narrative comments and survey of resident perception of feedback. PARTICIPANTS: Seventy internal medicine residents and their faculty evaluators at the University of Utah. APPROACH: Faculty narrative comments from 248 evaluations pre-and post-milestone implementation were analyzed for quality and Accreditation Council for Graduate Medical Education competency by area of strength and area for improvement. Seventy residents were surveyed regarding quality of feedback pre-and post-milestone implementation. KEY RESULTS: Qualitative analysis of narrative comments revealed nearly all evaluations pre-and postmilestone implementation included comments about areas of strength but were frequently vague and not related to competencies. Few evaluations included narrative comments on areas for improvement, but these were of higher quality compared to areas of strength (p < 0.001). Overall resident perception of quality of narrative comments was low and did not change following milestone implementation (p = 0.562) for the 86% of residents (N = 60/70) who completed the pre-and post-surveys. CONCLUSIONS: The quality of narrative comments was poor, and there was no evidence of improved quality following introduction of milestone-based evaluations. Comments on areas for improvement were of higher quality than areas of strength, suggesting an area for targeted intervention. Residents' perception of feedback quality did not change following implementation of milestonebased evaluations, suggesting that in the post-milestone era, internal medicine educators need to utilize additional interventions to improve quality of feedback.

ACGME Milestones in the Real World: A Qualitative Study Exploring Response Process Evidence

“…The ACGME intends for Milestones to be ''narratives, not numbers'' 21 ; meaningful narrative comments are the key to moving past average numerical assessment data. Although there are limitations to narrative comments, including that they may not cover the full breadth of subcompetencies 22 or may lack sufficient detail, 23 our participants noted that they can be helpful in corroborating or adjusting numerical scores.…”

Better Decision-Making: Shared Mental Models and the Clinical Competency Committee

“…28,43 Evidence that Milestones and entrustable professional activities (EPAs) provide mental models shared between assessors and learners is mixed. 33,[44][45][46][47][48][49] Part of the challenge is structural: Milestones and EPAs that are broadly defined are susceptible to variable interpretation. 49,50 This variability is affected by the larger challenge of the lack of an SMM of any instrument of assessment and the consequences of that assessment, which involves a process akin to negotiation.…”

“…49,50 This variability is affected by the larger challenge of the lack of an SMM of any instrument of assessment and the consequences of that assessment, which involves a process akin to negotiation. 45,46,51 The most important step in developing, maintaining, and using SMMs is to first acknowledge the complexity of a seemingly simple concept. Mental models are like Russian nesting dolls with each successive underlying model giving form to the next.…”

Reimagining Feedback for the Milestones Era

“…11,14 Facilitated Self-Reflection Simply providing assessment data, including the detailed Milestones narratives, to the trainee without conversation can be unhelpful and frustrating. 18,19 Kluger and DeNisi's review of feedback interventions demonstrated that feedback can be harmful or at best unproductive depending on the intervention. 20 Additional research from the health services domain revealed that providing quality data without support or guidance has minimal effects on behavioral change.…”