Associations between residency selection strategies and doctor performance: a meta-analysis

Abstract

OBJECTIVES The purpose of this study was to use meta-analysis to establish which of the information available to the resident selection committee is associated with resident or doctor performance.METHODS Multiple electronic databases were searched to 4 September 2012. Two reviewers independently selected studies that met the present inclusion criteria and extracted data in duplicate; disagreement was resolved by consensus. Risk for bias was assessed using a customised bias assessment tool. Measures of association were converted to a common effect size (Hedges' g). Meta-analysis was performed using the random-effects model for each selection strategy and all outcomes without pooling. Sensitivity analysis for each selection strategy-outcome pair was performed with pooling of effect size.RESULTS Eighty studies involving a total of 41 704 participants were included in the metaanalysis. Seventeen different selection strategies and 17 outcomes were assessed across these studies. The strongest positive associations referred to examination-based selection strategies, such as the US Medical Licensing Examination (USMLE) Step 1, and examination-based outcomes, such as scores on intraining examinations. Moderate positive associations were present for medical school marks and both examination-based and subjective outcomes. Minimal or no associations were seen for the selection tools represented by interviews, reference letters and deans' letters.CONCLUSIONS Standardised examination performance and medical school grades show the strongest associations with current measures of doctor performance. Deans' letters, reference letters and interviews all show a lower than expected strength of association given the relative value often assigned to them during resident doctor selection. Objective selection strategies are potentially the most useful to residency selection committees based on current evaluative methods. However, reports in the literature of validated long-term doctor performance outcomes are scant.

The association of USMLE Step 1 and Step 2 CK scores with residency match specialty and location

“…Particularly strong associations have been found between USMLE scores and other exam-related measures of success, such as scores on in-training examinations in specialties such as dermatology [7], emergency medicine [8], general surgery [9], and neurology and neurosurgery [10], as well as across specialties in meta-analyses [11,12]. Other positive associations were found between USMLE scores and a resident’s likelihood of successfully passing board examinations [12,13], especially on the first attempt [14,15]. On the other hand, USMLE scores tend to not be as strongly correlated with more subjective outcome measures, such as supervisor and faculty evaluations [10,11,15].…”

Use of the Interview in Resident Candidate Selection: A Review of the Literature

“…A recent metaanalysis of factors showed that examination-based selection strategies (eg, USMLE Step 1) had a strong positive association with in-service training examinations, whereas medical school grades had a less robust association with subjective outcomes such as resident performance evaluations. 7 USMLE Part 1 may even have a negative correlation with future clinical performance and professionalism. 8 Attributes such as leadership and professionalism are not assessable through standardized testing, but may be evident in a review of the curriculum vitae, personal statement, dean's letter, and letters of recommendation.…”

Behavioral Interviewing: Integrating ACGME Competency-Based Questions Into the Radiology Resident Selection Process

“…In this study, medical school grades had a less vigorous association with resident performance evaluations. 14 Other studies have shown that medical school performance does not predict the quality and performance of a resident. [16][17][18] In Brother's article, the USMLE Step 1 score actually suggested a negative correlation with future clinical performance and professionalism.…”