Assessing the Impact of Changes to USMLE Step 1 Grading on Evaluation of Neurosurgery Residency Applicants in the United States: A Program Director Survey

Stein, John S.; Estévez-Ordoñez, Dagoberto; Laskay, Nicholas M B; Atchley, Travis J.; Saccomano, Benjamin; Hale, Andrew T.; Patel, Om U; Burge, Kaitlin; Haynes, Will; Yadav, Ishant; Wagoner, Nicholas Van; Markert, James M.

doi:10.1016/j.wneu.2022.07.045

Cited by 18 publications

(11 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also intend to further explore applicability of this process to residency selection as well as use outcomes of our graduates in residency to better inform our current application process and which factors contribute to success across the continuum of medical education. [30][31][32]…”

Section: Discussionmentioning

confidence: 99%

Artificial Intelligence Screening of Medical School Applications: Development and Validation of a Machine-Learning Algorithm

Triola¹,

Reinstein²,

Marin³

et al. 2023

Academic Medicine

View full text Add to dashboard Cite

PurposeTo explore whether a machine-learning algorithm could accurately perform the initial screening of medical school applications. MethodUsing application data and faculty screening outcomes from the 2013 to 2017 application cycles (n = 14,555 applications), the authors created a virtual faculty screener algorithm. A retrospective validation using 2,910 applications from the 2013 to 2017 cycles and a prospective validation using 2,715 applications during the 2018 application cycle were performed. To test the validated algorithm, a randomized trial was performed in the 2019 cycle, with 1,827 eligible applications being reviewed by faculty and 1,873 by algorithm. ResultsThe retrospective validation yielded area under the receiver operating characteristic (AUROC) values of 0.83, 0.64, and 0.83 and area under the precision-recall curve (AUPRC) values of 0.61, 0.54, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The prospective validation yielded AUROC values of 0.83, 0.62, and 0.82 and AUPRC values of 0.66, 0.47, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The randomized trial found no significant differences in overall interview recommendation rates according to faculty or algorithm and among female or underrepresented in medicine applicants. In underrepresented in medicine applicants, there were no significant differences in the rates at which the admissions committee offered an interview (70 of 71 in the faculty reviewer arm and 61 of 65 in the algorithm arm; P = .14). No difference in the rate of the committee agreeing with the recommended interview was found among female applicants (224 of 229 in the faculty reviewer arm and 220 of 227 in the algorithm arm; P = .55). ConclusionsThe virtual faculty screener algorithm successfully replicated faculty screening of medical school applications and may aid in the consistent and reliable review of medical school applicants.

show abstract

Section: Discussionmentioning

confidence: 99%

Artificial Intelligence Screening of Medical School Applications: Development and Validation of a Machine-Learning Algorithm

Triola¹,

Reinstein²,

Marin³

et al. 2023

Academic Medicine

View full text Add to dashboard Cite

show abstract

“…After the initial search, 351 duplicates were removed and 44 were excluded after title and abstract review, 4,[7][8][9][10][11][12][13][14][15] based on inclusion and exclusion criteria. [16][17][18][19][20][21][22][23][24][25] Full-text review was performed on 98 articles, [26][27][28][29][30][31][32][33][34][35] of which 45 studies were included [36][37][38][39][40][41][42][43][44][45][46][47][48][49][50] (see Figure, Supplemental Digital Content 5, which illustrates a flow-chart of our literature search, as well as our full search strategies).…”

Section: Study Selectionmentioning

confidence: 99%

“…Thirty-four studies 4,[7][8][9][10][11][12][13][14][16][17][18][19][20][21][22][24][25][26][27][28][29][32][33][34][35][36]39,[41][42][43][44]46,49 included in the systematic review examined perceptions among PDs regarding the importance of Step 2 due to the score change, of which 12 studies 7,9,11,12,[19][20][21]24,28,29,32,41 qualified for the meta-analysis ( 3). Twenty-three studies 7,9-13,15-17,19-24,28,29,32,33,41,42,44,46 included in the systematic review examined perceptions among PDs regarding whether Step 2 will become required due to the score change, of which 12 studies 7,…”

Section: Medical Student Well-beingmentioning

confidence: 99%

Perceptions and Expectations Regarding the Step 1 Score Change Among Surgical and Non-Surgical Program Directors: A Systematic Review and Meta-Analysis

Srinivasan,

Samaan,

Premkumar

et al. 2024

The American Surgeon™

View full text Add to dashboard Cite

There are currently no studies examining differences in perceptions and expected impact of the Step 1 score change to pass/fail between surgical and non-surgical program directors (PDs). We conducted a systematic review in May 2023 of PubMed, Scopus, Web of Science, and PSYCInfo to evaluate studies examining PDs’ perspectives regarding the Step 1 score change. We performed random-effects meta-analyses to determine differences in perspectives among surgical and non-surgical PDs. Surgical PDs (76.8% [95% CI, 72.1%-82.0%], I2 = 52%) reported significantly greater rates of disagreement with the score change compared to non-surgical (65.1% [95% CI, 57.9%-73.1%], I2 = 69.7%) ( P = .01). Surgical PDs also reported significantly greater rates of agreement that the score change will increase the difficulty in objectively comparing applicants (88.1% [95% CI, 84.6%-91.7%], I2 = 16.4%), compared to non-surgical (81.0% [95% CI, 75.6%-86.8%], I2 = 72.6%) ( P = .04). There was less heterogeneity among non-surgical PDs (88.7% [95% CI, 86.2%-91.2%], I2 = 0%), compared to surgical (84.7% [95% CI, 79.0%-90.8%], I2 = 67.3%), regarding expected increases in emphasis on Step 2, although the difference in rates of agreement was not statistically significant. Overall, there is significant heterogeneity in the literature regarding expected changes in the residency application review process. Most PDs reported significant disagreement with the score change, greater expected difficulty in objectively evaluating applicants, and greater emphasis on Step 2, with surgical PDs reporting greater rates of disagreement, greater expected difficulty, and heterogeneity regarding expected increases in emphasis on Step 2, compared to non-surgical. Additionally, there is significant heterogeneity in the overall literature regarding expected changes in the residency application review process. Further research is needed to establish evidence-based guidelines that improve the overall residency application process for all stakeholders.

show abstract

“…Recent studies show that surgical program directors plan to increase the emphasis on Step 2 scores, with an anticipated emphasis on research quantity. 4–8…”

Section: Introductionmentioning

confidence: 99%

“…Recent studies show that surgical program directors plan to increase the emphasis on Step 2 scores, with an anticipated emphasis on research quantity. [4][5][6][7][8] This shift prompts the need for evidence-based guidance on Step examinations, as some students are choosing to take Step 2 before Step 1, and others are delaying Step 1. Our study aimed to characterize current Step examination order practices, explore factors influencing student preference, and assess their impact on Step scores amid changing USMLE grading.…”

Section: Introductionmentioning

confidence: 99%

Flipping Tradition on Its Head: A Single-Institution Study on United States Medical Licensing Examination Step Order

Zeng,

Harris,

Tian

et al. 2024

Plastic and Reconstructive Surgery - Global Open

View full text Add to dashboard Cite

Summary: Given the recent change in Step 1 grading, medical students are faced with decisions surrounding the optimal Step examination order and timing. There is a current lack of guidance outside commonly used United States Medical Licensing Examination study blogs. The aim of this study was to characterize current practices in Step examination order and understand the factors influencing student examination preference. To gather data, we used a 24-item survey and electronically distributed it through three separate email requests to medical students. Participants included medical students (MS1-4) attending a single institution, interested in a surgical residency. In total, 56 students responded to the survey (response rate of 20%). Most students (63.6%) opted to take Step 2 before Step 1. Justifications included the perceived importance of Step 2 for residency applications (33.3%), the change to pass/fail grading (27.0%), the relevance of Step 2 after rotations (27.0%), and a desire to obtain a Step 2 score earlier (7.9%). Step examination order did not lead to differences in Step 2 score (P = 0.459), time used to prepare for Step 1 (P = 0.396), or time used to prepare for Step 2 (P = 0.078). Due to the perceived importance of Step 2, most medical students at our institution who are interested in pursuing a surgical residency are reversing the traditional Step examination order. As this may represent a national trend, additional work needs to be done to provide guidance to medical students and identify the optimal step examination testing strategy.

show abstract

Assessing the Impact of Changes to USMLE Step 1 Grading on Evaluation of Neurosurgery Residency Applicants in the United States: A Program Director Survey

Cited by 18 publications

References 14 publications

Artificial Intelligence Screening of Medical School Applications: Development and Validation of a Machine-Learning Algorithm

Artificial Intelligence Screening of Medical School Applications: Development and Validation of a Machine-Learning Algorithm

Perceptions and Expectations Regarding the Step 1 Score Change Among Surgical and Non-Surgical Program Directors: A Systematic Review and Meta-Analysis

Flipping Tradition on Its Head: A Single-Institution Study on United States Medical Licensing Examination Step Order

Contact Info

Product

Resources

About