Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

Gottlieb, Michael; Jordan, Jaime; Siegelman, Jeffrey; Cooney, Robert; Stehman, Christine R.; Chan, Teresa M.

doi:10.1002/aet2.10519

Cited by 13 publications

(14 citation statements)

References 81 publications

(448 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, the qualifications section may benefit from revisions using validated workplace-based assessment scales including behavioral anchoring descriptions to ensure more consistent rating by evaluators. 7,15,16 Finally, our study found no strong correlation between the qualifications and grades. Previous research has reported grade and rank inflation to occur in up to 60% of SLOEs.…”

Section: Discussioncontrasting

confidence: 70%

The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

Miller

Krzyzaniak

Mannix

et al. 2021

AEM Education and Training

Self Cite

View full text Add to dashboard Cite

Objectives:The standardized letter of evaluation (SLOE) in emergency medicine (EM) is a widely used metric for determining interview invitations and ranking of candidates. Previous research has questioned the validity of certain sections of the SLOE.However, there remains a paucity of literature on the qualifications for EM section, which evaluates seven attributes of applicants. The aim of this study was to determine the correlation between the qualifications questions and grades, global assessment, and anticipated rank list position for EM applicants.Methods: A multi-institutional cross-sectional study was performed using SLOEs from applicants to three geographically distinct U.S. EM residency programs during the 2019-2020 application cycle. We abstracted EM rotation grade, qualifications scores, global assessment, and anticipated rank list position from the SLOEs. A Spearman correlation was calculated between each of the qualifications scores and the applicant's grades, global assessment, and anticipated rank list position in a pairwise fashion.

show abstract

Section: Discussioncontrasting

confidence: 70%

The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

Miller

Krzyzaniak

Mannix

et al. 2021

AEM Education and Training

Self Cite

View full text Add to dashboard Cite

show abstract

“…When collecting data, multiple studies emphasized the need for using both quantitative and qualitative components to avoid the limitations of purely numeric-based assessments. 17,44,50,52 Obtaining substantive comments on assessment forms can be challenging, 59,60 and future research should assess strategies for increasing qualitative comments. [61][62][63] Recent studies have also described tools for evaluating the quality of workplace-based assessment comments (e.g., Completed Clinical Evaluation Report Rating tool, Quality of Assessment for Learning score).…”

Section: Discussionmentioning

confidence: 99%

Workplace‐based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

Chan

Sebok‐Syer

Cheung

et al. 2020

AEM Education and Training

Self Cite

View full text Add to dashboard Cite

Objective: In the era of competency-based medical education (CBME), the collection of more and more trainee data is being mandated by accrediting bodies such as the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada. However, few efforts have been made to synthesize the literature around the current issues surrounding workplace-based assessment (WBA) data. This scoping review seeks to synthesize the landscape of literature on the topic of data collection and utilization for trainees' WBAs in emergency medicine (EM). Methods:The authors conducted a scoping review in the style of Arksey and O'Malley, seeking to synthesize and map literature on collecting, aggregating, and reporting WBA data. The authors extracted, mapped, and synthesized literature that describes, supports, and substantiates effective data collection and utilization in the context of the CBME movement within EM.Results: Our literature search retrieved 189 potentially relevant references (after removing duplicates) that were screened to 29 abstracts and papers relevant to collecting, aggregating, and reporting WBAs. Our analysis shows that there is an increasing temporal trend toward contributions in these topics, with the majority of the papers (16/29) being published in the past 3 years alone. Conclusion:There is increasing interest in the areas around data collection and utilization in the age of CBME.The field, however, is only beginning to emerge, leaving more work that can and should be done in this area.M cGaghie and colleagues first proposed competency-based medical education (CBME) in their World Health Organization report in 1978, 1 where they detailed the need to transition from a subjectbased curriculum or even an integrated curriculum to CBME in order to truly produce physicians that "can practice medicine at a defined level of proficiency, in accord with local conditions, to meet local needs." 1 The first step in transitioning to CBME (and realizing its full potential) is to define professional competence and its components. 1 In the late 1990s, medical education took this first step in the form of the Canadian Medical Education Directives for Specialists (Can-MEDS) roles 2,3 and the Accreditation Council for

show abstract

“…In the global shift towards competency‐based medical education (CBME), workplace‐based assessment (WBA) tools, informed by direct and indirect observation, have become the cornerstone of assessment 1 . While many of these WBA tools have been available for some time, including the mini‐clinical evaluation exercise (mini‐CEX), clinical encounter cards, direct observation of procedural skills (DOPS), and multi‐source feedback (MSF), 2,3 WBAs have historically been criticised for their low inter‐ and intra‐rater reliability, 4 and tendency to provide right‐shifted scores. Further, a recent systematic review of direct observation tools found that most lack strong validity evidence or reliability data 3 .…”

Section: Introductionmentioning

confidence: 99%

“…While many of these WBA tools have been available for some time, including the mini‐clinical evaluation exercise (mini‐CEX), clinical encounter cards, direct observation of procedural skills (DOPS), and multi‐source feedback (MSF), 2,3 WBAs have historically been criticised for their low inter‐ and intra‐rater reliability, 4 and tendency to provide right‐shifted scores. Further, a recent systematic review of direct observation tools found that most lack strong validity evidence or reliability data 3 . Thus, many programmes have taken the implementation of CBME as opportunity to refine WBA tools and processes.…”

Section: Introductionmentioning

confidence: 99%

Exploring the use of rating scales with entrustment anchors in workplace‐based assessment

et al. 2021

View full text Add to dashboard Cite

Purpose Competency‐based medical education (CBME) has prompted widespread implementation of workplace‐based assessment (WBA) tools using entrustment anchors. This study aimed to identify factors that influence faculty's rating choices immediately following assessment and explore their experiences using WBAs with entrustment anchors, specifically the Ottawa Surgical Competency Operating Room Evaluation scale. Method A convenience sample of 50 semi‐structured interviews with Emergency Medicine (EM) physicians from a single Canadian hospital were conducted between July and August 2019. All interviews occurred within two hours of faculty completing a WBA of a trainee. Faculty were asked what they considered when rating the trainee's performance and whether they considered an alternate rating. Two team members independently analysed interview transcripts using conventional content analysis with line‐by‐line coding to identify themes. Results Interviews captured interactions between 70% (26/37) of full‐time EM faculty and 86% (19/22) of EM trainees. Faculty most commonly identified the amount of guidance the trainee required as influencing their rating. Other variables such as clinical context, trainee experience, past experiences with the trainee, perceived competence and confidence were also identified. While most faculty did not struggle to assign ratings, some had difficulty interpreting the language of entrustment anchors, being unsure whether their assessment should be retrospective or prospective in nature, and if/how the assessment should change whether they were ‘in the room’ or not. Conclusions By going to the frontline during WBA encounters, this study captured authentic and honest reflections from physicians immediately engaged in assessment using entrustment anchors. While many of the factors identified are consistent with previous retrospective work, we highlight how some faculty consider factors outside the prescribed approach and struggle with the language of entrustment anchors. These results further our understanding of ‘in‐the‐moment’ assessments using entrustment anchors and may facilitate effective faculty development regarding WBA in CBME.

show abstract

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

Cited by 13 publications

References 81 publications

The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

Workplace‐based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

Exploring the use of rating scales with entrustment anchors in workplace‐based assessment

Contact Info

Product

Resources

About