Objective Nontechnical Skills Measurement Using Sensor-based Behavior Metrics in Surgical Teams

Abstract

Objective The purpose of this study was to identify objective measures that predict surgeon nontechnical skills (NTS) during surgery. Background NTS are cognitive and social skills that impact operative performance and patient outcomes. Current methods for NTS assessment in surgery rely on observation-based tools to rate intraoperative behavior. These tools are resource intensive (e.g., time for observation or manual labeling) to perform; therefore, more efficient approaches are needed. Method Thirty-four robotic-assisted surgeries were observed. Proximity sensors were placed on the surgical team and voice recorders were placed on the surgeon. Surgeon NTS was assessed by trained observers using the NonTechnical Skills for Surgeons (NOTSS) tool. NTS behavior metrics from the sensors included communication, speech, and proximity features. The metrics were used to develop mixed effect models to predict NOTSS score and in machine learning classifiers to distinguish between exemplar NTS scores (highest NOTSS score) and non-exemplar scores. Results NTS metrics were collected from 16 nurses, 12 assistants, 11 anesthesiologists, and four surgeons. Nineteen behavior features and overall NOTSS score were significantly correlated (12 communication features, two speech features, five proximity features). The random forest classifier achieved the highest accuracy of 70% (80% F1 score) to predict exemplar NTS score. Conclusion Sensor-based measures of communication, speech, and proximity can potentially predict NOTSS scores of surgeons during robotic-assisted surgery. These sensing-based approaches can be utilized for further reducing resource costs of NTS and team performance assessment in surgical environments. Application Sensor-based assessment of operative teams’ behaviors can lead to objective, real-time NTS measurement.

Surgical Sabermetrics

“…2, 3). Surgeon physiological measurements (n = 115) using noninvasive sensors were the preferred measurement of choice, but technology such as acoustic analysis (n = 3) [32][33][34] and movement (n = 4) 33,[35][36][37] were also employed. Cardiovascular measurements were the most common physiological metrics employed (n = 46), which include HRV, heart rate (HR), and blood pressure.…”

“…7,24,64,105,106 Social proximity is a predictor of behavior as it influences the opportunities available for effective teamwork and communication through nonverbal interactions and is suggestive of a shared mental model. 33,37 Studies have shown that computer vision techniques and proximity sensors for movement analysis can demonstrate an association between movement and teams with "poor" and "good" SA in the OR. Proximity sensors demonstrated the movement and closeness of the team around each other with "good SA" associated with restricted movements.…”

“…The time a team member spent close to the primary operating surgeon was a predictor of their NOTSS score. 33,37 Although no studies assessed decision-making directly, it is a complex and dynamic cognitive process that is intertwined with other nontechnical skills. 2,107 The effect of SA, stress, and increased CogL may cause detrimental effects on the human memory system, resulting in decision fatigue, tunnel-vision, and premature decision-making.…”

“…In addition, measurements such as EDA can indicate levels of task engagement but cannot distinguish whether the engagement was appropriate and task-relevant or inappropriate and at risk of degrading SA and subsequent performance. 88 Acoustic analysis can measure communication metrics relating to noise, including speech, 33,34 with Cha et al 33 noting a correlation between NOTSS scores and pitch, suggesting a link between speech metrics and leadership and teamwork behaviors. However, acoustic analysis cannot assess the content of speech.…”

Improving Healthcare Practice Through the Implementation of Human Factors and Ergonomics Principles

“…Cha and colleagues (Cha et al, 2022) used proximity sensors and voice recorders to investigate surgical teams’ nontechnical skills such as communication. In their paper, they provide preliminary evidence that support the efficacy of using Sensor-based measures of communication, speech, and proximity to train machine learning algorithms that may predict surgeons’ nontechnical skills.…”