Refining the evaluation of operating room performance

Abstract

Purpose: Accurate and consistent evaluation of resident operative performance is necessary but difficult to achieve. This study continues examination of the Southern Illinois University (SIU) operative performance rating system (OPRS) for previously unreported factors that may influence reliability, accuracy, and interpretability of results. Methods: OPRS evaluations of surgical residents by faculty at SIU, from 2004 to 2008, were analyzed for the most frequently rated procedures to determine (1) the elapsed time from the procedure until completion of rating, (2) patterns in responses of procedure-specific and global surgical skills items, and (3) whether particular evaluating surgeons differed in their stringency of ratings of resident operative performance. Results: Five-hundred sixty-six evaluations were analyzed: open colectomy (n ϭ 125), open inguinal hernia (n ϭ 103), laparoscopic cholecystectomy (n ϭ 199), and excisional biopsy (n ϭ 139). The number of residents evaluated per training level (PGY) ranged from 88 to 161. Median time to completion of evaluations was 11 days, 9 hours. The quickest evaluation was 18 hours after assignment, but most evaluations were completed within 4.5-22 days. Use of procedure-specific and global scale scores resulted in similar rank ordering of performances (single-measure intraclass correlation using the consistency model ϭ 0.88 [95% confidence interval ϭ 0.87-0.90]) and in similar absolute OPRS scores (0.89 [0.87-0.90]). Evaluating surgeons differed from one another in stringency of ratings across procedures (average difference ϭ 1.4 points on a 5-point scale). Resident ratings increased with PGY level for all 4 procedures.Conclusions: Substantial time elapses between a resident's performance in the operating room and the completion of the evaluation by faculty. This raises the question of whether surgeons remember the nuances of the procedure well enough to rate performance accurately. The item type used for rating operative performances does not affect the absolute rating assigned or the rank ordering of the performance. Differences in stringency of evaluators indicate the need for multiple resident performances to be observed by multiple surgeons. These findings are the foundation for an upcoming multi-institutional trial supported by the American Board of Surgery to examine the effects of elapsed time to evaluation and various resident characteristics when scoring operative performance. Novel hybrid Objective Structured Assessing Technique (OSAT)/ Objective Structured Clinical Examinations (OSCE) in comprehensive perioperative breast care: a three-year analysis of outcomes.Maker VK, Bonne S.

A Flexible Postoperative Debriefing Process Can Effectively Provide Formative Resident Feedback

“…These evaluations tend to focus on a limited number of criteria, lack a free text component, and primarily provide summative feedback. 11,12,[17][18][19][20][21][22] Although the briefing-intraoperativedebriefing (BID) helps to address many of these concerns, this model has its own set of limitations. 13 This model has residents set learning objectives for the case, followed by intraoperative and postoperative feedback, with debriefing from the supervising surgeon.…”

“…This is in comparison to previous work in which cases were selected for evaluation by educational administration. 11,12,26 The decision to allow resident selection of cases was made for technical reasons; a progressive implementation across our large program was anticipated to lead to greater compliance. It is our anecdotal experience that a relatively small number of faculty complete a disproportionate number of the PFFs, while the majority of attending surgeons complete just a few.…”

“…They often are focused on quantitative data and are occasionally filled out many days after the operative procedure. 11,12 Flexible approaches focusing on resident-defined learning objectives offer the potential to enter into a discussion, but may provide only limited and general feedback to residents, without critical resident self-assessment necessary for professional development. 13 The challenge of formative operative feedback is particularly acute in the surgery residency at Oregon Health & Science University (OHSU), one of the largest and most geographically diverse programs in the country.…”

Blinded Evaluation of Interrater Reliability of an Operative Competency Assessment Tool for Direct Laryngoscopy and Rigid Bronchoscopy

“…Unfortunately, evaluations performed at the end of a rotation may allow the evaluator to disregard the immediate techniquespecific feedback and instead provide more of a reflection on overall surgical competency. 8,9 The purpose of this study is to provide and test an operative competency assessment tool for otolaryngology residents completing rigid bronchoscopy and pediatric direct laryngoscopy procedures that would allow immediate, detailed evaluation of each resident's surgical performance. This reliability study used evaluators who were experienced but had no previous familiarity with the specific surgical residents.…”

Assessing Wire Navigation Performance in the Operating Room

“…These logs are used to establish competence, but fall short of measuring technical skill. 1 Only 31% of orthopaedic resident leaders feel that the case log volume should play a role in credentialing. 2 It is unclear what might replace case logs, but the president of the American Orthopaedic Association recently argued for competency-based training.…”