Predicting Performance at Graduation From Early ACGME Milestone Ratings: Longitudinal Learning Analytics in Professionalism and Communication in Vascular Surgery

Smith, Brigitte K.; Yamazaki, Kenji; Luman, Abigail; Tekian, Ara; Holmboe, Eric S.; Mitchell, Erica L.; Park, Yoon Soo; Hamstra, Stanley J.

doi:10.1016/j.jsurg.2022.09.002

Cited by 5 publications

(6 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…45,46 In more recent years, published studies explore instead medical practitioners' attitudes toward and engagement with eHealth data 47,52 or shed light on those systemic and cultural factors that influence the use of this type of data for CPD purposes. 48,51 Such topic evolution does not represent a novelty because it seems to be very much in line with the latest literature on predictive learning analytics, 53 doctor's attitudes toward CPD, 54,55 innovative life-long learning theoretical models, 56 and CPD strategic development approaches. 57 Nonetheless, its identification fills a research gap and can serve as a basis for further investigation in the field of PAH.…”

Section: Discussionmentioning

confidence: 95%

Using eHealth Data to Inform CPD for Medical Practitioners: A Scoping Review with a Consultation Exercise with International Experts

Pizzuti,

Palmieri,

Shaw

2023

J Contin Educ Health Prof

View full text Add to dashboard Cite

Introduction: eHealth data analytics is widely used in health care research. However, there is limited knowledge on the role of eHealth data analysis to inform continuing professional development (CPD). The aim of this study was to collate available research evidence on the use of eHealth data for the development of CPD programs and plans for medical practitioners. Methods: A scoping review was conducted using the six-stage Arksey and O'Malley Framework. A consultation exercise (stage 6) was performed with 15 international experts in the fields of learning and practice analytics to deepen the insights. Results: Scoping review. The literature searches identified 9876 articles published from January 2010 to May 2022. After screening and full-text review, a total of nine articles were deemed relevant for inclusion. The results provide varied—and at times partial or diverging—answers to the scoping review research questions. Consultation exercise. Research rigor, field of investigation, and developing the field were the three themes emerged from analysis. Participants validated the scoping review methodology and confirmed its results. Moreover, they provided a meta-analysis of the literature, a description of the current CPD ecosystem, and clear indications of what is and should be next for the field. Discussion: This study shows that there is no formal or well-established correlation between eHealth data and CPD planning and programming. Overall findings fill a gap in the literature and provide a basis for further investigation. More foundational work, multidisciplinary collaborations, and stakeholders' engagement are necessary to advance the use of eHealth data analysis for CPD purposes.

show abstract

Section: Discussionmentioning

confidence: 95%

Using eHealth Data to Inform CPD for Medical Practitioners: A Scoping Review with a Consultation Exercise with International Experts

Pizzuti,

Palmieri,

Shaw

2023

J Contin Educ Health Prof

View full text Add to dashboard Cite

show abstract

“…The ACGME Milestones reporting system was created in response to a recognized need for GME systems to have greater accountability to the public with regard to the competence of graduates as part of the transition to outcomes-based medical education . Longitudinal assessment data captured throughout training and across competency domains hold great promise to enable early detection of underperforming trainees and provide specific recommendations for remediation at actionable time points …”

Section: Discussionmentioning

confidence: 99%

Accreditation Council for Graduate Medical Education Milestone Training Ratings and Surgeons’ Early Outcomes

Smith,

Yamazaki,

Tekian

et al. 2024

JAMA Surg

Self Cite

View full text Add to dashboard Cite

ImportanceNational data on the development of competence during training have been reported using the Accreditation Council for Graduate Medical Education (ACGME) Milestones system. It is now possible to consider longitudinal analyses that link Milestone ratings during training to patient outcomes data of recent graduates.ObjectiveTo evaluate the association of in-training ACGME Milestone ratings in a surgical specialty with subsequent complication rates following a commonly performed operation, endovascular aortic aneurysm repair (EVAR).Design, Setting, and ParticipantsThis study of patient outcomes followed EVAR in the Vascular Quality Initiative (VQI) registry (4213 admissions from 208 hospitals treated by 327 surgeons). All surgeons included in this study graduated from ACGME-accredited training programs from 2015 through 2019 and had Milestone ratings 6 months prior to graduation. Data were analyzed from December 1, 2021, through September 15, 2023. Because Milestone ratings can vary with program, they were corrected for program effect using a deviation score from the program mean.ExposureMilestone ratings assigned to individual trainees 6 months prior to graduation, based on judgments of surgical competence.Main Outcomes and MeasuresSurgical complications following EVAR for patients treated by recent graduates during the index hospitalization, obtained using the nationwide Society for Vascular Surgery Patient Safety Organization’s VQI registry, which includes 929 participating centers in 49 US states.ResultsThe study included outcomes for 4213 patients (mean [SD] age, 73.25 [8.74] years; 3379 male participants [80.2%]). Postoperative complications included 9.5% major (400 of 4213 cases) and 30.2% minor (1274 of 4213 cases) complications. After adjusting for patient risk factors and site of training, a significant association was identified between individual Milestone ratings of surgical trainees and major complications in early surgical practice in programs with lower mean Milestone ratings (odds ratio, 0.50; 95% CI; 0.27-0.95).Conclusions and RelevanceIn this study, Milestone assessments of surgical trainees were associated with subsequent clinical outcomes in their early career. Although these findings represent one surgical specialty, they suggest Milestone ratings can be used in any specialty to identify trainees at risk for future adverse patient outcomes when applying the same theory and methodology. Milestones data should inform data-driven educational interventions and trainee remediation to optimize future patient outcomes.

show abstract

“…Interestingly, ICS emerged as a very strong predictor of first-time VCE pass, as one might expect given the oral presentation format of this examination. In vascular surgery, nearly 20% of trainees fail to reach competency in the professionalism and ICS Milestone competencies despite graduating from their programs, indicating some lingering uncertainty in how faculty assess these “newer” competencies 33 . The strong association of ICS assessment with a trainee’s performance on VCE is intriguing, nonetheless.…”

Section: Discussionmentioning

confidence: 99%

The Association of ACGME Milestones with Performance on American Board of Surgery Assessments

et al. 2023

Self Cite

View full text Add to dashboard Cite

Objective: To determine the relationship between, and predictive utility of, Milestone ratings and subsequent American Board of Surgery(ABS) vascular in-training(VSITE), qualifying(VQE), and certifying examination(VCE) performance in a national cohort of vascular surgery trainees. Summary Background Data: Specialty board certification is an important indicator of physician competence. However, predicting future board certification examination performance during training continues to be challenging. Methods: This is a national longitudinal cohort study examining relational and predictive associations between ACGME Milestone ratings and performance on VSITE, VQE, and VCE for all vascular surgery trainees from 2015 through 2021. Predictive associations between Milestone ratings and VSITE were conducted using cross-classified random-effects regression. Cross-classified random-effects logistic regression was utilized to identify predictive associations between Milestone ratings and VQE and VCE. Results: Milestone ratings were obtained for all residents and fellows(n=1,118) from 164 programs during the study period(July 2015-June 2021), including 145,959 total trainee assessments. Medical Knowledge(MK) and Patient Care(PC) Milestone ratings were strongly predictive of VSITE performance across all post-graduate years(PGYs) of training, with MK ratings demonstrating a slightly stronger predictive association overall(MK Coefficient 17.26-35.76,β = 0.15- 0.23). All core competency ratings were predictive of VSITE performance in PGYs 4 and 5. PGY 5 MK were highly predictive of VQE performance(OR 4.73,[95% CI: 3.87-5.78],P<0.001). PC subcompetencies were also highly predictive of VQE performance in the final year of training(OR 4.14,[95% CI: 3.17-5.41],P<0.001). All other competencies were also significantly predictive of first-attempt VQE pass with ORs of 1.53 and higher. PGY 4 ICS ratings(OR 4.0,[95% CI: 3.06-5.21],P<0.001) emerged as the strongest predictor of VCE first-attempt pass. Again, all subcompetency ratings remained significant predictors of first-attempt pass on CE with ORs of 1.48 and higher. Conclusions: ACGME Milestone ratings are highly predictive of future VSITE performance, and first-attempt pass achievement on VQE and VCE in a national cohort of surgical trainees.

show abstract

Predicting Performance at Graduation From Early ACGME Milestone Ratings: Longitudinal Learning Analytics in Professionalism and Communication in Vascular Surgery

Cited by 5 publications

References 26 publications

Using eHealth Data to Inform CPD for Medical Practitioners: A Scoping Review with a Consultation Exercise with International Experts

Using eHealth Data to Inform CPD for Medical Practitioners: A Scoping Review with a Consultation Exercise with International Experts

Accreditation Council for Graduate Medical Education Milestone Training Ratings and Surgeons’ Early Outcomes

The Association of ACGME Milestones with Performance on American Board of Surgery Assessments

Contact Info

Product

Resources

About