Rising to the Challenge: Residency Programs' Experience With Implementing Milestones-Based Assessment

Abstract

Background Changes to assessment efforts following the shift to milestones-based assessment in the ACGME Next Accreditation System have not been fully characterized.Objective This study describes themes in initial milestones-based assessment practices with the goal of informing continued implementation and optimization of milestones-based assessment.Methods Semistructured interviews were conducted with 15 residency program leaders in 6 specialties at 8 academic medical centers between August and December 2016. We explored what was retained, what was added, and what was changed from premilestones assessment efforts. We also examined the perceived impact of the shift to milestones-based assessment on the programs. Thematic analysis began after the first 5 interviews and ended once thematic sufficiency was reached. Two additional authors reviewed the codes, offered critical input, and informed the formation and naming of the final themes.Results Three themes were identified: (1) program leaders faced challenges to effective implementation; (2) program leaders focused on adaptability and making milestones work in what felt like a less than ideal situation for them; and (3) despite challenges, program leaders see value and utility in their efforts to move to milestones-based assessment. We describe a number of strategies that worked for programs during the transition, with perceived benefits acknowledged.Conclusions While adaptation to milestones has occurred and benefits are noted, negative impacts and challenges (eg, perceived lack of implementation guidance and faculty development resources) persist. There are important lessons learned (eg, utilizing implementation experiences formatively to improve curricula and assessment) in the transition to milestones-based assessment.

ACGME Milestones in the Real World: A Qualitative Study Exploring Response Process Evidence

“…We found that CCC members relied on assumptions about where an individual trainee should be based on time in the training program, rather than determining where they are performing. This practice is inconsistent with the intent of the Milestone framework and suggests that the ''benchmarking'' approach found by Dzara et al persists, 11 even after years of experience with the Milestones.…”

“…For example, in 2016, Dzara et al conducted an interview-based study of program directors across multiple specialties to gather information about Milestone implementation. Among the findings, participants indicated that some programs relied on ''a benchmark approach to assigning Milestones levels,'' assuming time-based achievement in the absence of complete assessment data, 11 signaling a flawed interpretation of, or approach to, Milestone ratings. Furthermore, 3 additional studies, with the aim of examining response process evidence, [12][13][14] retrospectively analyzed Milestone ratings for groups of residents, using quantitative analyses of large data sets to infer how raters use Milestones in resident assessment.…”

Current Milestones Assessment Practices, Needs, and Challenges of Program Directors: A Collective Case Study in a Pediatric Hospital Setting

“…Further, ensuring the consistency and alignment between a program curriculum and the Milestones assessment is vital knowledge for faculty to understand how trainees’ learning is progressing. Of course, other factors may also make the Milestone assessment difficult for faculty such as obtaining reliable data on certain milestones in unsupervised practice, lack of milestones implementation guidance, and knowledge deficiencies of faculty and trainees in the understanding of the meaning of the competencies in the context of their specialty or the right time and point for measurement of performance [ 5 , 18 ].…”

“…The need for ongoing professional development and faculty training is highly emphasized to overcome obstacles and barriers for successful implementation of Milestones in the literature [ 18 , 19 ]. The results from our study also suggest that PDs/APDs and faculty evaluators need to be trained in the science of assessment and evaluation as well as the use of survey’s technology platform which would make it easier to use with the existing Milestones practices and be able to track the progress of their trainees.…”

“…Due to the complexity involved in implementing Milestones, medical educators and program leaders have had obstacles and assessment challenges of the Milestones to assess medical trainees’ performance in various specialties [ 4 , 5 , 18 , 21 ]. The challenges included the need for direct observation, time constraints, faculty development, verbatim/language clarity, lack of guidance, new assessment tools, issues with measurement scales, adapting assessment efforts, and clarification of expectations.…”