Characterization of the Council of Emergency Medicine Residency Directors' Standardized Letter of Recommendation in 2011-2012

Abstract

Objectives: The Council of Emergency Medicine Residency Directors (CORD) introduced the standardized letter of recommendation (SLOR) in 1997, and it has become a critical tool for assessing candidates for emergency medicine (EM) training. It has not itself been evaluated since the initial studies associated with its introduction. This study characterizes current SLOR use to evaluate whether it serves its intended purpose of being standardized, concise, and discriminating.Methods: This retrospective, multi-institutional study evaluated letters of recommendation from U.S. allopathic applicants to three EM training programs during the 2011-2012 Electronic Residency Application Service (ERAS) application cycle. Distributions of responses to each question on the SLOR were calculated, and the free-text responses were analyzed. Two pilots, performed on five applicants each, assisted in developing a strategy for limiting interrater reliability.Results: Each of the three geographically diverse programs provided a complete list of U.S. allopathic applicants to their program. Upon randomization, each program received a list of coded applicants unique to their program randomly selected for data collection. The number of applicants was selected to reach a goal of approximately 200 SLORs per site (n = 602). Among this group, comprising 278 of 1,498 applicants (18.6%) from U.S. allopathic schools, a total of 1,037 letters of recommendation were written, with 724 (69.8%) written by emergency physicians. SLORs represented 57.9% (602/1037) of all LORs (by any kind of author) and 83.1% (602/724) of letters written by emergency physicians. Three hundred ninety-two of 602 SLORs had a single author (65.1%). For the question on "global assessment," students were scored in the top 10% in 234 of 583 of applications (40.1%; question not answered by some), and 485 of 583 (83.2%) of the applicants were ranked above the level of their peers. Similarly, >95% of all applicants were ranked in the top third compared to peers, for all but one section under "qualifications for emergency medicine." For 405 of 602 of all SLORs (67.2%), one or more questions were left unanswered, while 76 of all SLORs (12.6%) were "customized" or changed from the standard template. Finally, in 291 of 599 of SLORs (48.6%), the word count was greater than the recommended maximum of 200 words.Conclusions: Grade inflation is marked throughout the SLOR, limiting its ability to be discriminating. Furthermore, template customization and skipped questions work against the intention to standardize the SLOR. Finally, it is not uncommon for comments to be longer than guideline recommendations. As an assessment tool, the SLOR could be more discerning, concise, and standardized to serve its intended purpose.ACADEMIC EMERGENCY MEDICINE 2013; 20:926-932

Analysis of the Evaluative Components on the Standard Letter of Recommendation (SLOR) in Emergency Medicine

“…However, letter writers rely disproportionately on the top two categories rather than the full scale for assessment. Our findings are consistent with another recently published description of SLOR distribution of responses 7. There may be a number of explanations for this, including that students may only choose writers with whom they have an outstanding rapport, in effect maximizing the likelihood of an outstanding evaluation.…”

Does Applicant Gender Have an Effect on Standardized Letters of Evaluation Obtained During Medical Student Emergency Medicine Rotations?

“…1 Renamed the standardized letter of evaluation (SLOE) in 2013 2 , the intention of the SLOR/ SLOE was to make a standardized evaluation that could stratify students into quantiles based on performance during an emergency medicine (EM) rotation. 3 The SLOE replaces traditional narrative letters of recommendation for applicants to EM residency programs. Program directors (PDs) regard the SLOE as the most important item used for selecting applicants to interview.…”

Selection Of Endocrinology Subspecialty Trainees: Which Applicant Characteristics Are Associated With Performance During Fellowship Training?

“…[12] The conflicting results from these studies are likely due to a number of factors, including substantial variability in content of LoR, [13] different criteria used in grading LoR, [1] and difficulty in reaching a consensus on what constitutes a good LoR, [14] The shortcomings of the LoR have led some specialties including emergency medicine to create a standard letter of evaluation that might allow for meaningful comparisons to peers and a global perspective on an applicant’s candidacy. [15, 16] One study, however, reported that the global assessment score component of the standardized letter of recommendation correlated poorly with a candidate’s rank order position in the residency match. [17] Because of reported difficulty of grading the content of the LoR, we chose to simplify analysis of the LoR by only analyzing the presence of laudatory comparative statements, rather than a wide variety of exemplary statements and other qualitative factors.…”