Gender Differences Among Milestone Assessments in a National Sample of Pediatric Emergency Medicine Fellowship Programs

Abstract

Background: Understanding gender gaps in trainee evaluations is critical because these may ultimately determine the duration of training. Currently, no studies describe the influence of gender on the evaluation of pediatric emergency medicine (PEM) fellows. Objective:The objective of our study was to compare milestone scores of female versus male PEM fellows. Methods: This is a multicenter retrospective cohort study of a national sample of PEM fellows from July 2014 to June 2018. Accreditation Council for Medical Education (ACGME) subcompetencies are scored on a 5-point scale and span six domains: patient care (PC), medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal and communication skills (ICS). Summative assessments of the 23 PEM subcompetencies are assigned by each program's clinical competency committee and submitted semiannually for each fellow. Program directors voluntarily provided deidentified ACGME milestone reports. Demographics including sex, program region, and type of residency were collected. Descriptive analysis of milestones was performed for each year of fellowship. Multivariate analyses evaluated the difference in scores by sex for each of the subcompetencies.Results: Forty-eight geographically diverse programs participated, yielding data for 639 fellows (66% of all PEM fellows nationally); sex was recorded for 604 fellows, of whom 67% were female. When comparing the mean milestone scores in each of the six domains, there were no differences by sex in any year of training. When comparing scores within each of the 23 subcompetencies and correcting the significance level for comparison of multiple milestones, the scores for PC3 and ICS2 were significantly, albeit not meaningfully, higher for females.

Gender differences in emergency medicine resident assessment: A scoping review

“…Of the seven included studies that reported quantitative results across a wide range of different assessment types, 12,32,34–37 six did not identify any differences in scores between female and male residents 32,34–37 . The study by Dayal et al 12 was the only study to report statistically significant differences attributable to gender.…”

“…Seven of the 10 studies examined assessments based on the Accreditation Council for Graduate Medical Education (ACGME) EM or PEM milestones. 12,15,32,[35][36][37]39,40 Four studies investigated and reported the impact of assessor gender on resident performance scores, and none of the studies reported a statistically significant difference in scores between female and male assessors. 15,32,34,37 Only three studies included a priori definitions of meaningful differences in outcomes, 32,34,37 two of which included sample size calculations.…”

“…Overall, neither study found educationally significant differences in milestone scores between male and female residents/fellows, as the differences fell within the same "functional level" of a milestone rating and were therefore unlikely to represent an educationally meaningful difference or difference in ability. 35,36…”

“…While this study was robust in terms of its large sample across multiple centers, educationally meaningful outcome differences were not defined a priori to facilitate a clear interpretation of the identified differences in ACGME milestone scores. While the remaining six studies investigating quantitative score differences across multiple assessment types (simulation cases, procedures, and CCC outcomes) 32,[34][35][36][37] did not report any statistically significant differences in both singlecenter and multicenter settings, only three of these studies included an a priori definition of educationally meaningful outcome differences, 32,34,37 and only two reported sample size calculations. 32,34 Therefore, it is difficult know whether the remaining studies were sufficiently powered to detect differences attributable to gender.…”

“…While these findings do not discredit the existence of gender bias in EM, they do suggest that perhaps these workplace-based differences, when taken together with other forms of assessment data, may not impact downstream summative outcomes at the CCC level. 35,36 Previous studies have postulated that implicit bias and shifting role expectations may be factors influencing rater cognition 40 leading to gender bias in assessors' ratings of performance and written comments. Additionally, these factors may influence the design of workplace-based assessment tools further contributing to gender differences if, for example, the language of the rating scale anchors commend more agentic traits.…”