Letters of recommendations and personal statements for rhinology fellowship: A deep learning linguistic analysis

Vasan, Vikram; Cheng, Christopher P.; Lerner, David; Signore, Anthony Del; Schaberg, Madeleine R.; Govindaraj, Satish; Iloreta, Alfred Marc

doi:10.1002/alr.23153

Cited by 5 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This finding may indicate that less focus on work-related language and instead more about description of personal traits may be of more importance in an LOR. 11 The next phase of analysis was to determine the most accurate and precise predictive models of receiving an interview based on the prose and sentiment of LORs. CountVectorizer and TF-IDF were the best wordembedding techniques.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning for Predictive Analysis of Otolaryngology Residency Letters of Recommendation

Vasan,

Cheng,

Lerner

et al. 2024

The Laryngoscope

Self Cite

View full text Add to dashboard Cite

IntroductionLetters of recommendation (LORs) are a highly influential yet subjective and often enigmatic aspect of the residency application process. This study hypothesizes that LORs do contain valuable insights into applicants and can be used to predict outcomes. This pilot study utilizes natural language processing and machine learning (ML) models using LOR text to predict interview invitations for otolaryngology residency applicants.MethodsA total of 1642 LORs from the 2022–2023 application cycle were retrospectively retrieved from a single institution. LORs were preprocessed and vectorized using three different techniques to represent the text in a way that an ML model can understand written prose: CountVectorizer (CV), Term Frequency‐Inverse Document Frequency (TF‐IDF), and Word2Vec (WV). Then, the LORs were trained and tested on five ML models: Logistic Regression (LR), Naive Bayes (NB), Decision Tree (DT), Random Forest (RF), and Support Vector Machine (SVM).ResultsOf the 337 applicants, 67 were interviewed and 270 were not interviewed. In total, 1642 LORs (26.7% interviewed) were analyzed. The two best‐performing ML models in predicting interview invitations were the TF‐IDF vectorized DT and CV vectorized DT models.ConclusionThis preliminary study revealed that ML models and vectorization combinations can provide better‐than‐chance predictions for interview invitations for otolaryngology residency applicants. The high‐performing ML models were able to classify meaningful information from the LORs to predict applicant interview invitation. The potential of an automated process to help predict an applicant's likelihood of obtaining an interview invitation could be a valuable tool for training programs in the future.Level of EvidenceN/A Laryngoscope, 2024

show abstract

Section: Discussionmentioning

confidence: 99%

“…Other recent articles have evaluated bias in both SLORs and NLORs. [9][10][11] Studies have found that SLORs may lack discriminatory value that could be helpful in differentiating applicants. 10,[12][13][14] Hence, many letter writers submit additional narrative prose along with their SLORs, allowing more opportunity for analysis.…”

Section: Introductionmentioning

confidence: 99%

Machine Learning for Predictive Analysis of Otolaryngology Residency Letters of Recommendation

Vasan,

Cheng,

Lerner

et al. 2024

The Laryngoscope

Self Cite

View full text Add to dashboard Cite

show abstract

Gender‐based linguistic differences in letters of recommendation for rhinology fellowship over time: A dual‐institutional follow‐up study using natural language processing and deep learning

Vasan,

Cheng,

Edalati

et al. 2024

Int Forum Allergy Rhinol

View full text Add to dashboard Cite

show abstract

Optimizing the Evaluation of Letters of Recommendation for Obstetrics and Gynecology Residency Applicants

Morales-Ramirez,

Gutierrez,

DeMier

et al. 2024

O&G Open

View full text Add to dashboard Cite

OBJECTIVE: To develop and validate a ChatGPT-powered tool for assessing letters of recommendation in obstetrics and gynecology residency applicants. METHODS: This study was conducted in two phases at a university-based obstetrics and gynecology residency program. During the development phase, 30 letters of recommendation were used to create ChatGPT prompts based on a validated scoring rubric that assesses phrases, letter features, and applicant abilities. ChatGPT scores were compared with benchmarks using intraclass correlation coefficients (ICCs) to evaluate reliability, and ChatGPT processing times were measured against those of human reviewers. In the second phase (validation), 88 letters of recommendation from residents with and without academic deficiencies (n=30) were analyzed using the ChatGPT prompts developed and refined in the first stage. Residents with academic deficiencies (case group) were matched 1:1 with a control group of residents without noted deficiencies by medical school, gender, and training period. The letter of recommendation scores between the case and control groups then were compared using the Mann-Whitney U test. RESULTS: ChatGPT significantly reduced review time per letter, from 7.2 minutes to 45 seconds. Inter-rater reliability between ChatGPT and benchmark scores indicated strong agreement for specific phrases (ICC 0.82, 95% CI, 0.75–0.88), moderate agreement for letter features (ICC 0.75, 95% CI, 0.68–0.82), and fair agreement for applicant abilities (ICC 0.70, 95% CI, 0.62–0.78). After refinement, ICCs improved to 0.92 (95% CI, 0.88–0.95), 0.81 (95% CI, 0.74–0.87), and 0.77 (95% CI, 0.69–0.84). Mean letter of recommendation scores for specific phrases differed significantly between residents with deficiencies (4.49; 95% CI, 3.29–5.68) and residents in the control group (6.00; 95% CI, 5.00–7.00) (P=.017). Subgroup analysis showed an even greater gap between residents with multiple deficiencies (2.15; 95% CI, 1.42–3.38) and residents in the control group (6.00; 95% CI, 5.00–7.00; P<.001), suggesting a higher predictive value for at-risk residents. CONCLUSION: ChatGPT can enhance the residency selection process by providing reliable letter of recommendation analysis significantly faster than human reviewers and effectively identifying key phrases in letters of recommendation that could predict future academic deficiencies.

show abstract

Letters of recommendations and personal statements for rhinology fellowship: A deep learning linguistic analysis

Cited by 5 publications

References 12 publications

Machine Learning for Predictive Analysis of Otolaryngology Residency Letters of Recommendation

Machine Learning for Predictive Analysis of Otolaryngology Residency Letters of Recommendation

Gender‐based linguistic differences in letters of recommendation for rhinology fellowship over time: A dual‐institutional follow‐up study using natural language processing and deep learning

Optimizing the Evaluation of Letters of Recommendation for Obstetrics and Gynecology Residency Applicants

Contact Info

Product

Resources

About