Effectiveness of a focused educational intervention on resident evaluations from faculty

Abstract

OBJECTIVE:To improve the quality and specificity of written evaluations by faculty attendings of internal medicine residents during inpatient rotations. DESIGN:Prospective randomized controlled trial.SETTING: Four hospitals: tertiary care university hospital, Veterans' Administration hospital, and two community hospitals.PARTICIPANTS: Eighty-eight faculty and 157 residents from categorical and primary-care internal medicine residency training programs rotating on inpatient general medicine teams. INTERVENTION:Focused 20-minute educational session on evaluation and feedback, accompanied by 3 by 5 reminder card and diary, given to faculty at the start of their attending month. MEASUREMENTS AND MAIN RESULTS:Primary outcomes: 1) number of written comments from faculty specific to unique, preselected dimensions of competence; 2) number of written comments from faculty describing a specific resident behavior or providing a recommendation; and 3) resident Likert-scale ratings of the quantity and effect of feedback received from faculty. Faculty in the intervention group provided more written comments specific to defined dimensions of competence, a median of three comments per evaluation form versus two in the control group, but when adjusted for clustering by faculty, the difference was not statistically significant (P = .09). Regarding feedback, residents in the intervention group rated the quantity significantly higher (P = .04) and were significantly more likely to make changes in clinical management of patients than residents in the control group (P = .04). CONCLUSIONS:A brief, focused educational intervention delivered to faculty prior to the start of a ward rotation appears to have a modest effect on faculty behavior for written evaluations and promoted higher quality feedback given to house staff.

A Faculty Development Program to Reduce Rater Error on Milestone-Based Assessments

“…Participant faculty members were less likely to give residents ratings of 8 or 9 (reduced leniency error). Our findings are consistent with other research that shows rater training results in changes in how faculty rate residents when using the ABIM global rating form 13 and the Mini Clinical Evaluation Exercise (mini-CEX).…”

“…The use of videotaped scenarios in faculty development has been described in the literature as an effective teaching tool. 1,13 Frame-of-reference training was integrated and distributed among curriculum meetings, annual program review, and clinical competency committee meetings, during which examples and consistency of observable behaviors related to the Milestones were discussed and related to descriptive anchors on the rating scale. To be designated as a ''faculty development participant,'' an individual needed to attend at least 1 of the performance dimension training and 1 of the frame-ofreference training sessions.…”

The Quality of Written Feedback by Attendings of Internal Medicine Residents

“…3 Fortunately, studies have found that faculty development can modestly improve the quality of written and oral feedback. 8,32,37 Several specific recommendations emerge from this study that can help guide faculty development in providing feedback. First, faculty should understand the value of providing written comments that are multiple in number and scope.…”

Successful Implementation of a Direct Observation Program in an Ambulatory Block Rotation

“…10,11 Despite the central role and clear benefits of direct observation, its implementation in GME has proved challenging. [12][13][14][15][16] Studies indicate that little time is spent conducting direct observation during an average medical workday, 12 and a substantial percentage of residents state that they have never been directly observed doing certain important clinical tasks. 14,15 Barriers to frequent use include limited faculty time, faculty attention required toward clinical tasks, trainee reluctance to be observed, and concerns about the impact on the patient-resident relationship.…”