Concurrent Validity Evidence for Entrustable Professional Activities in General Surgery Residents

Abstract

BACKGROUND: A subset of Entrustable Professional Activities (EPAs) has been developed for general surgery. We aim to contribute validity evidence for EPAs as an assessment framework for general surgery residents, including concurrent validity compared to ACGME milestones, the current gold standard for evaluating competency. STUDY DESIGN: This is a cross-sectional study in a general surgery training program within a tertiary academic medical center. EPA assessments were submitted using a mobile app and scored on a numerical scale, mirroring milestones. EPA score distribution was analyzed with respect to post-graduate year (PGY) level and phase of care. Proportional odds logistic regression identified significant predictors. Spearman rank and Wilcoxon rank tests were used for comparisons with milestone ratings. RESULTS: From August 2018 to December 2019, 320 assessments were collected. EPA scores increased by PGY level. Operative phase EPA scores were significantly lower than nonoperative phase scores. PGY level, operative phase, and case difficulty significantly influenced entrustment scoring. EPA scores demonstrated strong correlation with nonoperative milestones patient care-1, medical knowledge-1, interpersonal and communication skills-2, interpersonal and communication skills-3, professionalism-1, professionalism-3, and practice-based learning and improvement-2 (ρ > 0.5, p < 0.05) and a weaker correlation with operative milestones patient care-3 and medical knowledge-2 (ρ < 0.5, p < 0.05). CONCLUSIONS: The influence of PGY level and operative phase on entrustment scoring supports the validity of EPAs as a formative evaluation framework for general surgery resident performance. In addition, evident correlations between EPA scores and respective milestone ratings provide concurrent validity evidence.

Disparities in the Operative Experience Between Female and Male General Surgery Residents

“…In addition, with a push toward entrustment as the new paradigm of surgical training, faculty must be intentional about entrusting female trainees with more operative autonomy 31. It is promising that 2 recent publications found that entrustable professional activity evaluations may limit sex bias in resident evaluations 32,33. Programs should also consider intentional case allocation and frequent audits of case logs to ensure equitable distribution and adequate time to correct deficiencies.…”

“…Female residents also performed fewer thoracic [32(23-48) vs 37, (25-55) P < 0.01], trauma [27 (18-40) vs 31 (20-47), P = 0.01], and vascular [102 (78-133) vs 113 (86-151), P < 0.01] cases compared with male residents. In contrast, female residents performed more breast [53 (42-71) vs 49 (39-64), P < 0.01] and anorectal cases [33 (25-44) vs 30 (22)(23)(24)(25)(26)(27)(28)(29)(30)(31)(32)(33)(34)(35)(36)(37)(38)(39)(40)(41), P < 0.01] than male residents.…”

“…31 It is promising that 2 recent publications found that entrustable professional activity evaluations may limit sex bias in resident evaluations. 32,33 Programs should also consider intentional case allocation and frequent audits of case logs to ensure equitable distribution and adequate time to correct deficiencies. For example, our time-trend analysis indicates that starting in 2015, there was a significant upswing in female resident surgeon junior cases, which coincides with when the ACGME implemented its new requirement for residents to complete at least 250 cases by the end of the second year of residency.…”

Entrustable Professional Activities in Surgery

“…[20][21][22] In 2 single-institution studies evaluating all 5 pilot EPAs and the gallbladder EPA, respectively, study authors also demonstrated strong positive correlation between EPA microassessment ratings and ACGME Milestone scores. 20,23 These correlations make intuitive sense given the deliberate mapping of Milestone competencies to EPAs by the ABS EPA development groups, while also providing evidence for their validity compared with current forms of resident assessment. 17 In related work, Chen et al 24 described the development and testing of 6 procedure-specific assessments they termed surgical EPAs based on the ABS Operative Performance Assessment framework.…”

“…39 This is supported by both the intentional mapping of EPAs to 5 to 7 subcompetencies and associated milestones during their development, along with data from some confirmatory studies bolstering this correlation. 17,18,20 As EPAs are increasingly adopted within a program, program leadership may be able to reduce the end-of-rotation evaluation burden to the minimum information needed to supplement Milestone ratings, as the majority of residents' performance assessments would have been captured through EPA microassessments. 39 Importantly, while EPAs will serve as an important data source to inform clinical competency committee ratings of Milestones, not Created in collaboration with relevant society groups, these Fellowship Council EPAs can be customized based on the subtype of gastrointestinal surgery fellowship, with topics that move beyond core general surgery residency EPAs to more advanced activities that a fellowship-trained surgeon would be expected to manage (eg, "Evaluate and manage patients with parastomal hernias," "Evaluate and manage patients with esophageal motility disorders").…”