Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Abstract

IMPORTANCE US internal medicine residency programs are now required to rate residents using milestones. Evidence of validity of milestone ratings is needed.OBJECTIVE To compare ratings of internal medicine residents using the pre-2015 resident annual evaluation summary (RAES), a nondevelopmental rating scale, with developmental milestone ratings. DESIGN, SETTING, AND PARTICIPANTSCross-sectional study of US internal medicine residency programs in the 2013-2014 academic year, including 21 284 internal medicine residents (7048 postgraduate-year 1 [PGY-1], 7233 PGY-2, and 7003 PGY-3).EXPOSURES Program director ratings on the RAES and milestone ratings. MAIN OUTCOMES AND MEASURESCorrelations of RAES and milestone ratings by training year; correlations of medical knowledge ratings with American Board of Internal Medicine (ABIM) certification examination scores; rating of unprofessional behavior using the 2 systems. RESULTSCorresponding RAES ratings and milestone ratings showed progressively higher correlations across training years, ranging among competencies from 0.31 (95% CI, 0.29 to 0.33) to 0.35 (95% CI, 0.33 to 0.37) for PGY-1 residents to 0.43 (95% CI, 0.41 to 0.45) to 0.52 (95% CI, 0.50 to 0.54) for PGY-3 residents (all P values <.05). Linear regression showed ratings differed more between PGY-1 and PGY-3 years using milestone ratings than the RAES (all P values <.001). Of the 6260 residents who attempted the certification examination, the 618 who failed had lower ratings using both systems for medical knowledge than did those who passed (RAES difference, −0.9; 95% CI, −1.0 to −0.8; P < .001; milestone medical knowledge 1 difference, −0.3; 95% CI, −0.3 to −0.3; P < .001; and medical knowledge 2 difference, −0.2; 95% CI, −0.3 to −0.2; P < .001). Of the 26 PGY-3 residents with milestone ratings indicating deficiencies on either of the 2 medical knowledge subcompetencies, 12 failed the certification examination. Correlation of RAES ratings for professionalism with residents' lowest professionalism milestone ratings was 0.44 (95% CI, 0.43 to 0.45; P < .001).CONCLUSIONS AND RELEVANCE Among US internal medicine residents in the 2013-2014 academic year, milestone-based ratings correlated with RAES ratings but with a greater difference across training years. Both rating systems for medical knowledge correlated with ABIM certification examination scores. Milestone ratings may better detect problems with professionalism. These preliminary findings may inform establishment of the validity of milestone-based assessment.

ACGME Milestones Within Subspecialty Training Programs: One Institution's Experience

“…Specifically, in a study assessing internal medicine residency milestones over a 3-year period, the authors noted an increase from 2.46 to 3.92 over a 36-month period of training across all milestones. 20 A similar trend was reported in national Internal Medicine Milestone ratings, 9 which showed an increase up to 0.77 for each PGY level, highlighting a much greater change per PGY than within our cohort of fellows. While recent data from the ACGME show upward trends of milestone ratings in subspecialty training programs nationwide, 17 this does not account for myriad institutional factors that affect milestone ratings.…”

“…Early experiences with the residency milestones provided ample validity evidence in this population. [9][10][11][12][13][14][15][16] One study supported the construct validity of competency assessment using milestones through the demonstration of a predictable upward trend of milestone ratings over the course of residency training. 10 While recent data from the ACGME highlighted a similar upward trend in subspecialty training programs nationally, 17 it is not clear if this trend would be observed within researchoriented fellowship training programs.…”

Parental Leave in Graduate Medical Education: Recommendations for Reform

“…25 Early data on the impact of the milestones shows that they successfully measure GME outcomes. 26,27 The ACGME Clinical Learning Environment Review monitors fatigue management and wellness. 28…”

The Journey to Competency-based Medical Education - Implementing Milestones

“…1,3,16 Early validity research studies are encouraging in supporting the use of Milestones professional development. [6][7][8][9][10] As a sufficient number of residents graduate and enter practice we will be able to examine the links, or associations, between Milestone performance and quality of practice: the ultimate outcome goal of the Milestone initiative.…”