Regular Formal Evaluation Sessions are Effective as Frame-of-Reference Training for Faculty Evaluators of Clerkship Medical Students

Abstract

BACKGROUND: Face-to-face formal evaluation sessions between clerkship directors and faculty can facilitate the collection of trainee performance data and provide frame-of-reference training for faculty. OBJECTIVE: We hypothesized that ambulatory faculty who attended evaluation sessions at least once in an academic year (attendees) would use the Reporter-Interpreter-Manager/Educator (RIME) terminology more appropriately than faculty who did not attend evaluation sessions (non-attendees). DESIGN: Investigators conducted a retrospective cohort study using the narrative assessments of ambulatory internal medicine clerkship students during the 2008-2009 academic year. PARTICIPANTS: The study included assessments of 49 clerkship medical students, which comprised 293 individual teacher narratives. MAIN MEASURES: Single-teacher written and transcribed verbal comments about student performance were masked and reviewed by a panel of experts who, by consensus, (1) determined whether RIME was used, (2) counted the number of RIME utterances, and (3) assigned a grade based on the comments. Analysis included descriptive statistics and Pearson correlation coefficients. KEY RESULTS: The authors reviewed 293 individual teacher narratives regarding the performance of 49 students. Attendees explicitly used RIME more frequently than non-attendees (69.8 vs. 40.4 %; p< 0.0001). Grades recommended by attendees correlated more strongly with grades assigned by experts than grades recommended by non-attendees (r=0.72; 95 % CI (0.65, 0.78) vs. 0.47; 95 % CI (0.26, 0.64); p= 0.005). Grade recommendations from individual attendees and non-attendees each correlated significantly with overall student clerkship clinical performance [r=0.63; 95 % CI (0.54, 0.71) vs. 0.52 (0.36, 0.66), respectively], although the difference between the groups was not statistically significant (p=0.21). CONCLUSIONS: On an ambulatory clerkship, teachers who attended evaluation sessions used RIME terminology more frequently and provided more accurate grade recommendations than teachers who did not attend. Formal evaluation sessions may provide frame-of-reference training for the RIME framework, a method that improves the validity and reliability of workplace assessment.

Faculty development for the evaluation system: a dual agenda

“…While performance data should be reviewed annually, alternating topics or performing exercises to improve the quality of evaluations they submit are fun ways to keep faculty interested in the process. This dual agenda faculty development model provides a practical framework that can translate into more effective evaluations from your faculty, increase self-awareness of evaluation tendencies, and facilitate self-directed changes in teaching behaviors 7,8,16…”

“…Simply familiarizing them with the evaluation instrument is not sufficient. Prior work has centered on frame-of-reference training to calibrate faculty assessments to the grading scale 3–7. Additionally, faculty can receive instruction on quality assessment language or the vocabulary that should be used to build an effective narrative statement.…”

Grading Practices and Distributions Across Internal Medicine Clerkships

“…At the clerkship level, we should strive for a greater number of evaluations and a greater variety of assessments, as well as enhanced faculty development. The use of regular formal evaluation sessions with a structured framework of assessment (RIME) has demonstrated efficacy in improving the validity of the grading process, [10][11][12][13] It has been argued that developing a reliable clinical grading system for medical students to use across institutions would not be feasible, or worth the effort. 28,29 Indeed, there is clearly a need for more data about how useful any particular grade is in determining the ultimate competence of the physician.…”

Defining and Assessing the 21st-Century Physician in Training

“…3 The tasks for today's educators are to define the competencies that appropriately describe the 21st-century physician most capable of providing high-quality care in an increasingly complex system, and then to design methods that accurately and reliably assess those competencies in a way that promotes deep and meaningful learning among physician trainees. The articles in this issue by Hauer et al 4 and Hemmer et al 5 outline steps for achieving these tasks. Hauer and colleagues provide a roadmap that uses a rigorous approach to defining EPAs for medical students.…”