ACGME Milestones Within Subspecialty Training Programs: One Institution's Experience

Abstract

Background The Accreditation Council for Graduate Medical Education Milestones were created as a criterion-based framework to promote competency-based education during graduate medical education. Despite widespread implementation across subspecialty programs, extensive validity evidence supporting the use of milestones within fellowship training is lacking.Objective We assessed the construct and response process validity of milestones in subspecialty fellowship programs in an academic medical center.Methods From 2014-2016, we performed a single center retrospective cohort analysis of milestone data from fellows across 5 programs. We analyzed summary statistics and performed multivariable linear regression to assess change in milestone ratings by training year and variability in ratings across fellowship programs. Finally, we examined a subset of Professionalism and Interpersonal and Communication Skills subcompetencies from the first 6 months of training to identify the proportion of fellows deemed ''ready for independent practice'' in these domains. ResultsMilestone data were available for 68 fellows, with 75 933 unique subcompetency ratings. Multivariable linear regression, adjusted for subcompetency and subspecialty, revealed an increase of 0.17 (0.16-0.19) in ratings with each postgraduate year level increase (P , .005), as well as significant variation in milestone ratings across subspecialties. For the Professionalism and Interpersonal and Communication Skills domains, mean ratings within the first 6 months of training were 3.78 and 3.95, respectively. ConclusionsWe noted a minimal upward trend of milestone ratings in subspecialty training programs, and significant variability in implementing milestones across differing subspecialties. This may suggest possible difficulties with the construct validity and response process of the milestone system in certain medical subspecialties.

Milestone Level Changes From Residency to Fellowship: A Multicenter Cohort Study

“…The 11 subcompetencies included a combination of ''context-independent'' and ''context-dependent'' subcompetencies. 23 Context-independent subcompetencies are those in which the assessment of Milestone levels should not change based on the context of the training environment. As noted by Heath et al, the competencies of professionalism and interpersonal and communication skills are examples of context-independent subcompetencies.…”

“…As noted by Heath et al, the competencies of professionalism and interpersonal and communication skills are examples of context-independent subcompetencies. 23 In contrast, the subcompetencies of patient care and medical knowledge are examples of contextdependent subcompetencies for which the assessment of Milestone levels may change based on the context of the training environment. 23 Other subcompetencies in systems-based practice and practice-based learning and improvement are more ambiguous.…”

“…23 In contrast, the subcompetencies of patient care and medical knowledge are examples of contextdependent subcompetencies for which the assessment of Milestone levels may change based on the context of the training environment. 23 Other subcompetencies in systems-based practice and practice-based learning and improvement are more ambiguous. 23 There were no identical interpersonal and communication skills subcompetencies between general pediatrics residency and NPM fellowship.…”

Discriminatory ability of milestones: An analysis of milestone variability by obstetrics and gynecology subspecialty

“…Interprogram variability has been reported, as has variability between specialties. 9,10 In terms of milestone accuracy within a particular evaluation, Beeson et al evaluated the rate of "straight line scoring" (SLS, defined as a resident being assigned the same score across milestone subcompetencies) and showed that a small but meaningful number of programs submitted SLS ratings. Because of the statistical improbability of SLS, any SLS ratings reduce the validity assertions of the milestone assessments.…”

Miles to Go Before We Sleep. Reforming the Pulmonary and Critical Care Milestones to Improve Trainee Assessment

“…To ensure they met the needs of physicians, trainees, and society at large, the ACGME committed to ongoing data collection and review, with the opportunity for future revisions. Feedback from educators, program directors, and trainees, with additional input from qualitative and quantitative research, highlighted the benefits and challenges of the original Milestones within the subspecialties ( 3 , 5 , 10 , 11 ). In PCCM and other fields, this system appeared to achieve construct validity based on the year-by-year upward trajectory of milestone ratings among fellows ( Figure 3 ) ( 12 ).…”