Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making

Abstract

Background Videos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment. Methods A multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird’s-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment. Results and discussion The scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio–video quality, and questionable feasibility for assessing readiness for practice. Conclusion Video-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio–video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training.

Correcting for Rater Effects in Operating Room Surgical Skills Assessment

“…VBA of surgical skill is an emerging methodology, which can be used to avoid some sources of rater effects 18 . An assessment of anesthesia trainees found that VBA yielded comparable scores with real‐time assessment, 19 and an assessment of orthopedic trainees found good inter‐rater reliability for VBA by expert surgeons 20 . VBA allows for anonymous rating when trainee attributes are masked from the rater.…”

“…18 An assessment of anesthesia trainees found that VBA yielded comparable scores with real-time assessment, 19 and an assessment of orthopedic trainees found good inter-rater reliability for VBA by expert surgeons. 20 VBA allows for anonymous rating when trainee attributes are masked from the rater. It minimizes the impact of implicit biases and may reduce the impact of raters' leniency or strictness with respect to trainee attributes.…”