BACKGROUND:
Holistic review of applications may optimize recruitment of residents by seeking out characteristics best aligned with program culture. The goals of this mixed methods research were to engage residency recruitment stakeholders to develop a holistic scoring rubric, measure the correlation between the rubric score and the final global rating used to rank applicants for the National Resident Matching Program Match, and qualitatively analyze committee discussions at the end of the interview day about applicants for potential unconscious biases.
METHODS:
Forty stakeholders (32 faculty, 3 chief residents, and 5 administrative staff) completed an iterative consensus-driven process to identify the most highly valued applicant attributes, and a corresponding standardized question for each attribute. The rubric was used after the interview and after the group discussion to score all 203 applicants (29% underrepresented in medicine, 55% male) interviewed virtually during 1 recruitment season. Committee discussions of the day’s candidates (15 separate interview days) were transcribed and analyzed using a phenomenological approach to identify biases.
RESULTS:
The final rubric included 10 dimensions: interpersonal attributes, scholarship, leadership, resilience, medical knowledge, medical school performance (excluding test scores), community service, mature learner, motivation for anesthesiology, and diversity. The first 5 dimensions were given equal weight, while the next 4 had lower but equal weighting among them. Diversity received the lowest weight overall. The mean rubric score (max 36) equaled 25.92 (standard deviation [SD] 1.99, median 26, range 13–29), which was significantly correlated (r = 0.94, P < .001) with the final global rating (mean = 4.35 SD 0.29, range 2.25–4.9) used for ranking. The United States Medical Licensing Examination (USMLE) scores, underrepresented in medicine status, geographic region of the applicant, and gender were not correlated with the global rating. Interrater reliability among 32 committee members was high (r = 0.77, 95% confidence interval [CI], 0.73–0.80). Thematic analysis of 4079 coded text segments identified 9 major bias types, with the most common being: in-group bias for candidates perceived as being similar to typical residents currently in the program, stereotyping via opinions of the candidate’s personality as being a good fit for the specialty, cohort bias comparing an applicant to other applicants that interview day instead of the entire season, and anchoring bias due to the interviewer’s initial impression of the candidate’s motivation to become an anesthesiologist.
CONCLUSIONS:
Stakeholder-driven holistic review that more broadly emphasizes an applicant’s experiences and attributes can be successfully implemented in evaluating residency applicants. Committee discussions revealed various biases that warrant further investigation and mitigation strategies.
