The Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs

Hauer, Karen E.; Clauser, Jerome C.; Lipner, Rebecca S.; Holmboe, Eric S.; Caverzagie, Kelly J.; Hamstra, Stanley J.; Hood, Sarah; Iobst, William; Warm, Eric J.; McDonald, Furman S.

doi:10.7326/m15-2411

Cited by 43 publications

(49 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As previously reported, 2814 residents (1569 PGY-1, 888 PGY-2, and 357 PGY-3) had at least 1 milestone subcompetency rated as not assessable. 6 Demographic characteristics are shown in Table 1 and eTable 4 in the Supplement.…”

Section: Resultsmentioning

confidence: 99%

“…The RAES system was intended for summative purposes, whereas milestone ratings have been initially used more formatively. 6 Items on rating forms may not align with behaviors or qualities that supervisors observe or how supervisors think. [7][8][9] Ratings of trainee performance tend to be inflated, clustered at the high end of the scale, and based on a limited number of trainee characteristics.…”

Section: Discussionmentioning

confidence: 99%

“…In a national cross-sectional study from postgraduate year (PGY) 1 to PGY-3, US internal medicine milestone-based ratings differed by residents' training year. 6 Pediatric milestone ratings within a single training program also stratified residents by training year, whereas nonmilestone-based ratings did not. 7 It is critically important to assess the validity of any new assessment.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Hauer

Vandergrift

Hess³

et al. 2016

JAMA

Self Cite

View full text Add to dashboard Cite

IMPORTANCE US internal medicine residency programs are now required to rate residents using milestones. Evidence of validity of milestone ratings is needed.OBJECTIVE To compare ratings of internal medicine residents using the pre-2015 resident annual evaluation summary (RAES), a nondevelopmental rating scale, with developmental milestone ratings. DESIGN, SETTING, AND PARTICIPANTSCross-sectional study of US internal medicine residency programs in the 2013-2014 academic year, including 21 284 internal medicine residents (7048 postgraduate-year 1 [PGY-1], 7233 PGY-2, and 7003 PGY-3).EXPOSURES Program director ratings on the RAES and milestone ratings. MAIN OUTCOMES AND MEASURESCorrelations of RAES and milestone ratings by training year; correlations of medical knowledge ratings with American Board of Internal Medicine (ABIM) certification examination scores; rating of unprofessional behavior using the 2 systems. RESULTSCorresponding RAES ratings and milestone ratings showed progressively higher correlations across training years, ranging among competencies from 0.31 (95% CI, 0.29 to 0.33) to 0.35 (95% CI, 0.33 to 0.37) for PGY-1 residents to 0.43 (95% CI, 0.41 to 0.45) to 0.52 (95% CI, 0.50 to 0.54) for PGY-3 residents (all P values <.05). Linear regression showed ratings differed more between PGY-1 and PGY-3 years using milestone ratings than the RAES (all P values <.001). Of the 6260 residents who attempted the certification examination, the 618 who failed had lower ratings using both systems for medical knowledge than did those who passed (RAES difference, −0.9; 95% CI, −1.0 to −0.8; P < .001; milestone medical knowledge 1 difference, −0.3; 95% CI, −0.3 to −0.3; P < .001; and medical knowledge 2 difference, −0.2; 95% CI, −0.3 to −0.2; P < .001). Of the 26 PGY-3 residents with milestone ratings indicating deficiencies on either of the 2 medical knowledge subcompetencies, 12 failed the certification examination. Correlation of RAES ratings for professionalism with residents' lowest professionalism milestone ratings was 0.44 (95% CI, 0.43 to 0.45; P < .001).CONCLUSIONS AND RELEVANCE Among US internal medicine residents in the 2013-2014 academic year, milestone-based ratings correlated with RAES ratings but with a greater difference across training years. Both rating systems for medical knowledge correlated with ABIM certification examination scores. Milestone ratings may better detect problems with professionalism. These preliminary findings may inform establishment of the validity of milestone-based assessment.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Hauer

Vandergrift

Hess³

et al. 2016

JAMA

Self Cite

View full text Add to dashboard Cite

show abstract

“…25 Early data on the impact of the milestones shows that they successfully measure GME outcomes. 26,27 The ACGME Clinical Learning Environment Review monitors fatigue management and wellness. 28…”

Section: Fmlamentioning

confidence: 99%

Parental Leave in Graduate Medical Education: Recommendations for Reform

Vassallo

Jeremiah

Leanne

et al. 2019

The American Journal of Medicine

View full text Add to dashboard Cite

AAIM is the largest academically focused specialty organization representing departments of internal medicine at medical schools and teaching hospitals in the United States and Canada. As a consortium of five organizations, AAIM represents department chairs and chiefs; clerkship, residency, and fellowship program directors; division chiefs; and academic and business administrators as well as other faculty and staff in departments of internal medicine and their divisions.

show abstract

“…[9][10][11][12][13][14][15][16] One study supported the construct validity of competency assessment using milestones through the demonstration of a predictable upward trend of milestone ratings over the course of residency training. 10 While recent data from the ACGME highlighted a similar upward trend in subspecialty training programs nationally, 17 it is not clear if this trend would be observed within researchoriented fellowship training programs. Furthermore, generic milestones may not be sufficient for diverse medical subspecialties, particularly given significant variability in training program structure and possible variability in milestone interpretation.…”

Section: Introductionmentioning

confidence: 97%

ACGME Milestones Within Subspecialty Training Programs: One Institution's Experience

Heath

Dine

2019

Journal of Graduate Medical Education

View full text Add to dashboard Cite

Background The Accreditation Council for Graduate Medical Education Milestones were created as a criterion-based framework to promote competency-based education during graduate medical education. Despite widespread implementation across subspecialty programs, extensive validity evidence supporting the use of milestones within fellowship training is lacking.Objective We assessed the construct and response process validity of milestones in subspecialty fellowship programs in an academic medical center.Methods From 2014-2016, we performed a single center retrospective cohort analysis of milestone data from fellows across 5 programs. We analyzed summary statistics and performed multivariable linear regression to assess change in milestone ratings by training year and variability in ratings across fellowship programs. Finally, we examined a subset of Professionalism and Interpersonal and Communication Skills subcompetencies from the first 6 months of training to identify the proportion of fellows deemed ''ready for independent practice'' in these domains. ResultsMilestone data were available for 68 fellows, with 75 933 unique subcompetency ratings. Multivariable linear regression, adjusted for subcompetency and subspecialty, revealed an increase of 0.17 (0.16-0.19) in ratings with each postgraduate year level increase (P , .005), as well as significant variation in milestone ratings across subspecialties. For the Professionalism and Interpersonal and Communication Skills domains, mean ratings within the first 6 months of training were 3.78 and 3.95, respectively. ConclusionsWe noted a minimal upward trend of milestone ratings in subspecialty training programs, and significant variability in implementing milestones across differing subspecialties. This may suggest possible difficulties with the construct validity and response process of the milestone system in certain medical subspecialties.

show abstract

The Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs

Abstract: None.

Cited by 43 publications

References 22 publications

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Parental Leave in Graduate Medical Education: Recommendations for Reform

ACGME Milestones Within Subspecialty Training Programs: One Institution's Experience

Contact Info

Product

Resources

About