Implementation of the Operating Room Black Box Research Program at the Ottawa Hospital Through Patient, Clinical, and Organizational Engagement: Case Study

Abstract

Background A large proportion of surgical patient harm is preventable; yet, our ability to systematically learn from these incidents and improve clinical practice remains limited. The Operating Room Black Box was developed to address the need for comprehensive assessments of clinical performance in the operating room. It captures synchronized audio, video, patient, and environmental clinical data in real time, which are subsequently analyzed by a combination of expert raters and software-based algorithms. Despite its significant potential to facilitate research and practice improvement, there are many potential implementation challenges at the institutional, clinician, and patient level. This paper summarizes our approach to implementation of the Operating Room Black Box at a large academic Canadian center. Objective We aimed to contribute to the development of evidence-based best practices for implementing innovative technology in the operating room for direct observation of the clinical performance by using the case of the Operating Room Black Box. Specifically, we outline the systematic approach to the Operating Room Black Box implementation undertaken at our center. Methods Our implementation approach included seeking support from hospital leadership; building frontline support and a team of champions among patients, nurses, anesthesiologists, and surgeons; accounting for stakeholder perceptions using theory-informed qualitative interviews; engaging patients; and documenting the implementation process, including barriers and facilitators, using the consolidated framework for implementation research. Results During the 12-month implementation period, we conducted 23 stakeholder engagement activities with over 200 participants. We recruited 10 clinician champions representing nursing, anesthesia, and surgery. We formally interviewed 15 patients and 17 perioperative clinicians and identified key themes to include in an information campaign run as part of the implementation process. Two patient partners were engaged and advised on communications as well as grant and protocol development. Many anticipated and unanticipated challenges were encountered at all levels. Implementation was ultimately successful, with the Operating Room Black Box installed in August 2018, and data collection beginning shortly thereafter. Conclusions This paper represents the first step toward evidence-guided implementation of technologies for direct observation of performance for research and quality improvement in surgery. With technology increasingly being used in health care settings, the health care community should aim to optimize implementation processes in the best interest of health care professionals and patients.

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill

“…Deep learning in particular is well suited for tackling technical skills assessment due to its robustness to noise and flexibility to learn an optimal feature set representative of task performance from large, unstructured, and multi-modal data sources. Further, new innovations allow for the collection of large-scale multi-modal data in previously unwelcoming environments, such as the operating room [ 3 ].…”

Creating a Practical Transformational Change Management Model for Novel Artificial Intelligence–Enabled Technology Implementation in the Operating Room

“…In this article, we present a case of recent transformational change management (CM) in the ORs of our institution that did not fit the existing CM models ( Table 1 ), 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 which were identified from a narrative literature search. 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 The lack of fit was partly because many existing tools were not geared toward organizational or transformational change but instead focused on individual, developmental, or transitional changes.…”

“…It was developed to assist translation of scientific advances into practice, although has been proven slow and inequitable. 18 , 19 The purpose of RE-AIM is not aligned with our project’s needs: to manage change associated with implementing novel technology in the OR. Differing from RE-AIM’s requirement of concurrent measurement of all dimensions of the of model, the HNT model intends to guide CMs via a journey map that leads the project teams to focus on one or a few specific aspects of the project at a time and in a sequence.…”

“…This specific AI–enabled surgical analytical system has been applied to develop diagnostic and predictive intraoperative algorithms. 18 , 19 , 22 It improves upon the advantages of current surgical data recording through (1) objective intraoperative data collection, presentation, human performance analysis, and review 18 , 23 , 24 , 25 ; (2) postoperative team debriefing enhancement 18 ; (3) improved nontechnical performance analysis 26 , 27 ; and (4) enhanced understanding of adverse events. 23 It promises opportunities to automate, synchronize, and ultimately analyze multiple continuous complex data streams (eg, device-related patient data such as vital signs and intraoperative data: surgical site and room audiovisual recordings) arising from the OR environment in short periods of time for purposes such as nonbiased and evidence-based adverse event review and panoramic observation of OR efficiency and teamwork.…”

‘You can’t do quality between surgical cases and tea time’: barriers to surgeon engagement in quality improvement

“…Strategies to empower surgeons to trust and use individual-level feedback include choosing common metrics that can be tracked automatically, developing a plan for sharing data and improving outcomes with hospitals and administrators, and the use of electronic medical records and dashboards to provide real-time feedback 5. Emerging technologies such as the operating room Black Box Research Program15 and robotic-assisted surgery16 may also provide novel strategies for technical improvement, including the use of objective measurements of surgical performance and the impact on patient outcomes and surgical safety.…”