Values-Based Resident Selection in an Internal Medicine-Pediatrics Residency Program

Drum, Benjamin; Lamb, Sara; Gradick, Casey

doi:10.1007/s11606-022-07857-y

Cited by 6 publications

(7 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All applications had been reviewed in depth by the internal medicine–pediatrics program director using the holistic review values-based rubric described previously and sorted into not invited or invited for interview (IFI) without input from the MLM. 8 The number of annotations per value and the total number of value label annotations were quantified for each application and analyzed with respect to interview status using a 2-tailed, unpaired t test.…”

Section: Approachmentioning

confidence: 99%

“…We recently used a modified Delphi method to identify a set of resident values important for success in our internal medicine–pediatrics residency program: academic strength; intellectual curiosity; compassion; communication; work ethic; teamwork; leadership; self-awareness; diversity, equity, and inclusion (DEI); professionalism; and adaptability. 8 Incorporation of these values into our ranking system resulted in uncoupling of our ranking with United States Medical Licensing Examination Step 2 scores. However, values-based ranking was time-intensive and thus limited to interviewed applicants.…”

Section: Problemmentioning

confidence: 99%

“…The creation and evaluation of our applicant value framework have been described previously. 8 Briefly, we used a modified Delphi approach that consisted of literature review, structured interviews with local stakeholders, and iterative cycles with local experts to derive 11 values important for success in the internal medicine–pediatrics residency program of the University of Utah. These values were also mapped to the Accreditation Council for Graduate Medical Education core competencies.…”

Section: Approachmentioning

confidence: 99%

See 2 more Smart Citations

Using Natural Language Processing and Machine Learning to Identify Internal Medicine–Pediatrics Residency Values in Applications

et al. 2023

Self Cite

View full text Add to dashboard Cite

Problem: Although holistic review has been used successfully in some residency programs to decrease bias, such review is time-consuming and unsustainable for many programs without initial prescreening. The unstructured qualitative data in residency applications, including notable experiences, letters of recommendation, personal statement, and medical student performance evaluations, require extensive time, resources, and metrics to evaluate; therefore, previous applicant screening relied heavily on quantitative metrics, which can be socioeconomically and racially biased. Approach: Using residency applications to the University of Utah internal medicine–pediatrics program from 2015 to 2019, the authors extracted relevant snippets of text from the narrative sections of applications. Expert reviewers annotated these snippets into specific values (academic strength; intellectual curiosity; compassion; communication; work ethic; teamwork; leadership; self-awareness; diversity, equity, and inclusion; professionalism; and adaptability) previously identified as associated with resident success. The authors prospectively applied a machine learning model (MLM) to snippets from applications from 2023, and output was compared with a manual holistic review performed without knowledge of MLM results. Outcomes: Overall, the MLM had a sensitivity of 0.64, specificity of 0.97, positive predictive value of 0.62, negative predictive value of 0.97, and F1 score of 0.63. The mean (SD) total number of annotations per application was significantly correlated with invited for interview status (invited: 208.6 [59.1]; not invited: 145.2 [57.2]; p < .001). In addition, 8 of the 10 individual values were significantly predictive of an applicant’s invited for interview status. Next Steps: The authors created an MLM that can identify several values important for resident success in internal medicine–pediatrics programs with moderate sensitivity and high specificity. The authors will continue to refine the MLM by increasing the number of annotations, exploring parameter tuning and feature engineering options, and identifying which application sections have the highest correlation with invited for interview status.

show abstract

Section: Approachmentioning

confidence: 99%

Section: Problemmentioning

confidence: 99%

Section: Approachmentioning

confidence: 99%

See 1 more Smart Citation

Using Natural Language Processing and Machine Learning to Identify Internal Medicine–Pediatrics Residency Values in Applications

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…[16][17][18][19][20][21][22] Manuscripts vary in whether they focus on predictions of success in the specialty overall [20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35] or specifically within training programs. 16,18,19,[36][37][38][39][40][41] Authors of manuscripts in this category describe priority being given to selecting those who will do well during postgraduate training and also focus on avoiding those who will need remediation or be disruptive to program leadership. 8,16,29,31,37,[42][43][44][45][46] For example, Burkhardt et al reflected this dichotomy, while focusing on academic success, by stating:…”

Section: Using Available Metrics To Predict Performance In Postgradua...mentioning

confidence: 99%

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

Caretta-Weyer,

Eva,

Schumacher

et al. 2023

Academic Medicine

View full text Add to dashboard Cite

Purpose: The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature in order to explore practical implications and present a research agenda. Method: Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion. Results: Five distinct perspectives or value statements were identified in the thematic analysis: (1) Utilizing available metrics to predict performance in postgraduate training; (2) Identifying the best applicants via competitive comparison; (3) Seeking alignment between applicant and program in the selection process; (4) Ensuring diversity, mitigation of bias, and equity in the selection process; and (5) Optimizing the logistics or mechanics of the selection process. Conclusions: This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

show abstract

“…Approach : Previously, we used a modified Delphi method to identify and internally validate 11 values of successful residents for our internal medicine-pediatrics residency program (MP) in the Mountain West region: academic strength; intellectual curiosity; compassion; communication; work ethic; teamwork; leadership; self-awareness; diversity, equity, and inclusion (DEI); professionalism; and adaptability. 1 Using residency applications to our program from 2015 to 2021, we created a program to extract relevant snippets of text pertaining to these values. These snippets were then annotated by expert reviewers and assigned to specific values.…”

mentioning

confidence: 99%

Using Natural Language Processing to Identify Key Values in Internal Medicine-Pediatrics Residency Applications

Drum,

Shi,

Peterson

et al. 2023

Academic Medicine

Self Cite

View full text Add to dashboard Cite

Values-Based Resident Selection in an Internal Medicine-Pediatrics Residency Program

Cited by 6 publications

References 33 publications

Using Natural Language Processing and Machine Learning to Identify Internal Medicine–Pediatrics Residency Values in Applications

Using Natural Language Processing and Machine Learning to Identify Internal Medicine–Pediatrics Residency Values in Applications

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

Using Natural Language Processing to Identify Key Values in Internal Medicine-Pediatrics Residency Applications

Contact Info

Product

Resources

About