Integrating machine learning predictions for perioperative risk management: Towards an empirical design of a flexible-standardized risk assessment tool

Abraham, Joanna; Bartek, Brian; Meng, Alicia; King, Christopher R.; Xue, Bing; Lu, Chenyang; Avidan, Michael S.

doi:10.1016/j.jbi.2022.104270

Cited by 19 publications

(6 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, some user needs depend on the specific use case. When showing predictions at a single time point at the end of surgery, anesthesiology clinicians have preferred tabular formats, 25 whereas graphical formats were desired in this study to see trends over time. Intraoperative prediction models for hypotension and hypoxemia have incorporated graphs into their display interfaces, 9,10 but these studies did not directly ask users about their display preferences.…”

Section: Discussionmentioning

confidence: 99%

User-Centered Design of a Machine Learning Dashboard for Prediction of Postoperative Complications

Fritz¹,

Pugazenthi²,

Budelier³

et al. 2023

Anesthesia &Amp; Analgesia

Self Cite

View full text Add to dashboard Cite

BACKGROUND: Machine learning models can help anesthesiology clinicians assess patients and make clinical and operational decisions, but well-designed human-computer interfaces are necessary for machine learning model predictions to result in clinician actions that help patients. Therefore, the goal of this study was to apply a user-centered design framework to create a user interface for displaying machine learning model predictions of postoperative complications to anesthesiology clinicians. METHODS: Twenty-five anesthesiology clinicians (attending anesthesiologists, resident physicians, and certified registered nurse anesthetists) participated in a 3-phase study that included (phase 1) semistructured focus group interviews and a card sorting activity to characterize user workflows and needs; (phase 2) simulated patient evaluation incorporating a low-fidelity static prototype display interface followed by a semistructured interview; and (phase 3) simulated patient evaluation with concurrent think-aloud incorporating a high-fidelity prototype display interface in the electronic health record. In each phase, data analysis included open coding of session transcripts and thematic analysis. RESULTS: During the needs assessment phase (phase 1), participants voiced that (a) identifying preventable risk related to modifiable risk factors is more important than nonpreventable risk, (b) comprehensive patient evaluation follows a systematic approach that relies heavily on the electronic health record, and (c) an easy-to-use display interface should have a simple layout that uses color and graphs to minimize time and energy spent reading it. When performing simulations using the low-fidelity prototype (phase 2), participants reported that (a) the machine learning predictions helped them to evaluate patient risk, (b) additional information about how to act on the risk estimate would be useful, and (c) correctable problems related to textual content existed. When performing simulations using the high-fidelity prototype (phase 3), usability problems predominantly related to the presentation of information and functionality. Despite the usability problems, participants rated the system highly on the System Usability Scale (mean score, 82.5; standard deviation, 10.5). CONCLUSIONS: Incorporating user needs and preferences into the design of a machine learning dashboard results in a display interface that clinicians rate as highly usable. Because the system demonstrates usability, evaluation of the effects of implementation on both process and clinical outcomes is warranted. (Anesth Analg 2024;138:804-13) KEY POINTS• Question: What design considerations are necessary to create a machine learning dashboard that anesthesiology clinicians find useful and usable to help them in order to estimate patient risk for postoperative complications? • Findings: A user interface with electronic health record integration, explanations that permit identification of modifiable predictors, and a simple layout can result in high usability rating...

show abstract

Section: Discussionmentioning

confidence: 99%

User-Centered Design of a Machine Learning Dashboard for Prediction of Postoperative Complications

Fritz¹,

Pugazenthi²,

Budelier³

et al. 2023

Anesthesia &Amp; Analgesia

Self Cite

View full text Add to dashboard Cite

show abstract

“…To assist with prioritizing cases for review and quantifying patient risks for ACT-OR communication, a web application displayed machine learning predictions of individual patient risk of several major adverse events (Supplemental Figure 2). [18][19][20][21] The implementation and validation of this web application is described elsewhere. 22 ACT staff also used the Epic EHR to access patient information.…”

Section: Intervention: Anesthesiology Control Tower (Act)mentioning

confidence: 99%

Effect of Telemedicine Support for Intraoperative Anaesthesia Care on Postoperative Outcomes: The TECTONICS Randomised Clinical Trial

King,

Fritz,

Gregory

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Novel applications of telemedicine can improve care quality and patient outcomes. Telemedicine for intraoperative decision support has not been rigorously studied. Methods: This single centre randomised clinical trial (clinicaltrials.govNCT03923699) of unselected adult surgical patients was conducted between July 1, 2019 and January 31, 2023. Patients received usual care or decision support from a telemedicine service, the Anesthesiology Control Tower (ACT). The ACT provided real-time recommendations to intraoperative anaesthesia clinicians based on case reviews, machine-learning forecasting, and physiologic alerts. ORs were randomised 1:1. Co-primary outcomes of 30-day all-cause mortality, respiratory failure, acute kidney injury (AKI), and delirium were analysed as intention-to-treat. Results: The trial completed planned enrolment with 71927 surgeries (35956 ACT; 35971 usual care). After multiple testing correction, there was no significant effect of the ACT vs. usual care on 30-day mortality [641/35956 (1.8%) vs 638/35971 (1.8%), risk difference 0.0% (95% CI -0.2% to 0.3%), p=0.96], respiratory failure [1089/34613 (3.1%) vs 1112/34619 (3.2%), risk difference -0.1% (95% CI -0.4% to 0.3%), p=0.96], AKI [2357/33897 (7%) vs 2391/33795 (7.1%), risk difference -0.1% (-0.6% to 0.4%), p=0.96], or delirium [1283/3928 (32.7%) vs 1279/3989 (32.1%), risk difference 0.6% (-2.0% to 3.2%), p=0.96]. There were no significant differences in secondary outcomes or in sensitivity analyses. Conclusions: In this large RCT of a novel application of telemedicine-based remote monitoring and decision support using real-time alerts and case reviews, we found no significant differences in postoperative outcomes. Large-scale intraoperative telemedicine is feasible, and we suggest future avenues where it may be impactful.

show abstract

“…Throughout this paper, we adopt Rubin's potential outcomes model [25] and consider the population of COVID-19 subjects where each subject 𝑖 is associated with a 𝑝-dimensional feature 𝑋 𝑖 ∈ 𝑋 ⊆ R 𝑝 , a binary treatment assignment indicator 𝑊 𝑖 ∈ {0, 1}, and two potential outcomes 𝑌 𝑖 (1), 𝑌 𝑖 (0) ∈ {0, 1} drawn from a Bernoulli distribution (𝑌 𝑖 (1), 𝑌 𝑖 (0))|𝑋 𝑖 ∼ 𝑃 (.|𝑋 𝑖 ). For an observational dataset 𝐷 comprising 𝑛 independent samples of the tuple {𝑋 𝑖 ,𝑊 𝑖 , 𝑌 𝑖 (𝑊 𝑖 )}, where 𝑌 𝑖 (𝑊 𝑖 ) and 𝑌 𝑖 (1 − 𝑊 𝑖 ) are the factual and the counterfactual outcomes, respectively, we are interested in the probability of treatment assignment (propensity score) 𝑝 (𝑥) = 𝑃 (𝑊 𝑖 = 1|𝑋 𝑖 ), the potential outcome with treatment E[𝑌 𝑖 (1)|𝑋 𝑖 ] and the potential outcome without treatment E[𝑌 𝑖 (0)|𝑋 𝑖 ]. As the treatment outcomes are binary (survival or death), the treatment is "impactful" only if treatment leads to a change from death to survival.…”

Section: Problem Formulationmentioning

confidence: 99%

“…Developing an ECMO decision-assistant model differs from typical supervised machine learning problems or standard treatment effect problems in healthcare. Compared with a supervised clinical problem [1,17,19,20,36,46,47,49], the entire vector of treatment effects can never be obtained, but only the factual outcomes aligned with the individualized treatment assignments. Compared with a typical treatment effect problem [25,39], it faces the challenges of strong selection bias, scarcity of treatment cases and curse of dimensionality.…”

Section: Introductionmentioning

confidence: 99%

Assisting Clinical Decisions for Scarcely Available Treatment via Disentangled Latent Representation

Xue

Said

et al. 2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Self Cite

View full text Add to dashboard Cite

Extracorporeal membrane oxygenation (ECMO) is an essential lifesupporting modality for COVID-19 patients who are refractory to conventional therapies. However, the proper treatment decision has been the subject of significant debate and it remains controversial about who benefits from this scarcely available and technically complex treatment option. To support clinical decisions, it is a critical need to predict the treatment need and the potential treatment and no-treatment responses. Targeting this clinical challenge, we propose Treatment Variational AutoEncoder (TVAE), a novel approach for individualized treatment analysis. TVAE is specifically designed to address the modeling challenges like ECMO with strong treatment selection bias and scarce treatment cases. TVAE conceptualizes the treatment decision as a multi-scale problem. We model a patient's potential treatment assignment and the factual and counterfactual outcomes as part of their intrinsic characteristics that can be represented by a deep latent variable model. The factual and counterfactual prediction errors are alleviated via a reconstruction regularization scheme together with semi-supervision, and the selection bias and the scarcity of treatment cases are mitigated by the disentangled and distribution-matched latent space and the label-balancing generative strategy. We evaluate TVAE on

show abstract

Integrating machine learning predictions for perioperative risk management: Towards an empirical design of a flexible-standardized risk assessment tool

Cited by 19 publications

References 45 publications

User-Centered Design of a Machine Learning Dashboard for Prediction of Postoperative Complications

User-Centered Design of a Machine Learning Dashboard for Prediction of Postoperative Complications

Effect of Telemedicine Support for Intraoperative Anaesthesia Care on Postoperative Outcomes: The TECTONICS Randomised Clinical Trial

Assisting Clinical Decisions for Scarcely Available Treatment via Disentangled Latent Representation

Contact Info

Product

Resources

About