Reliability of Performance‐based Clinical Skill Assessment of Emergency Medicine Residents

Abstract

Objective: To test the overall reliability of a performance-based clinical skill assessment for entering emergency medicine (EM) residents. Also, to investigate the reliability of separate reporting of diagnostic and management scores for a standardized patient case, subjective scoring of patient notes, and interstation exercise scores. Methods: Thirty-four first-year EM residents were tested using a 10-station standardized patient (SP) examination. Following each 10-minute encounter, the residents completed a patient note that included differential diagnosis and management. The residents also were asked to read an ECG or chest x-ray (CXR) associated with each case. History, physical examination, and interpersonal skills were scored by the SPs. The patient note, CXR, and ECG readings were scored by faculty emergency physicians. Intercase reliability was determined for the residents. amination in the 1970s as part of the certification process, aimed at better assessing clinical competence. In addition, the ABEM required residency directors to attest to the clinical skills of applicants to the Board. In response to that requirement, we are now developing a comprehensive performance-based assessment that will provide baseline data about entering emergency medicine (EM) residents' global clinical competence and competence in related skills, with specific information about their strengths and weaknesses.Encounters with standardized patients (SPs)-nonphysicians trained to portray a specific history and physical examination-along with related interstation exercises have been used in undergraduate and graduate medical education to assess clinical competence.' Traditionally, history, physical examination, interpersonal skills, and the patient note written after the SP encounter have been the major components separately reported in performance-based assessment programs due to their relatively acceptable reliabilit~.~ The reliability of additional components is still being investigated.

Emergency Radiology “Boot Camp”: Educating Emergency Medicine Residents Using E‐learning Radiology Modules

“…The diagnostic tasks included the interpretation of chest radiographs, which are one of the most commonly performed radiologic examinations in the ED setting. Of all the clinical and diagnostic tasks that they were required to perform, they scored the lowest on chest x‐ray interpretation …”

Decision Making by Emergency Room Physicians and Residents

“…In this study, the more experienced staff physicians' inputs represent the benchmark to which residents' values are compared. This approach is widely used in the literature to evaluate performance of less experienced clinicians and can take the form of comparing novices to experts performing the same task (Nodine, Kundel, Mello-Thoms, Weinstein, Orel, Sullivan & Conant, 1999;Sklar, Hauswald & Johnson, 1991), or having expert clinicians evaluating the performance of novice clinicians (Burdick et al, 1996;Steinbach, 2002;Wray & Friedland, 1983). As a measure of proper elicitation, we use a level of agreement beyond chance between values for CDSS input variables provided by staff physicians and residents.…”

Protected Clinical Teaching Time and a Bedside Clinical Evaluation Instrument in an Emergency Medicine Training Program

“…Our CEE was developed after publication of the ACGME general competencies and the ''Model of the Clinical Practice of Emergency Medicine,'' and incorporates language and principles from these important documents. Other studies involving emergency medicine faculty have noted that standardized objective evaluations provide better interrater reliability than global assessment scoring, 8,9 and we have tried to incorporate that lesson as well.…”