Bridging the implementation gap of machine learning in healthcare

Seneviratne, Martin; Shah, Nigam H.; Chu, Larry F.

doi:10.1136/bmjinnov-2019-000359

Cited by 118 publications

(86 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite considerable improvements in measures of performance of predictive models in healthcare, there have been relatively few successes using such models to provide better clinical outcomes at lower costs 39,40 . We believe that this is in part because the translation of modeling advances into improvements in clinical care requires integrating the model's output into complex human workflows that are separate from model performance 6,8,9 . If the actions taken in response to a predictive model are embedded in such a system, the ability to make predictions, and improvements in predictive accuracy alone, are not sufficient to improve care.…”

Section: Discussionmentioning

confidence: 99%

“…Predictive models, which estimate the probability of some event of interest occuring in a specified time frame in the future, have been developed for events such as heart failure, inpatient mortality, and patient deterioration 5 . However, there have been relatively few success stories where these models led to impact on what matters to patients, providers, and healthcare decision makers such as, reduction in costs, lower rate of those clinical events, and increased access to care 6,7 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A framework for making predictive models useful in practice

Jung

Kashyap

Avati

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Objective To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. Materials and Methods We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included non-clinical reasons that make ACP inappropriate, limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP. Results Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care. Discussion The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit. Conclusion An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A framework for making predictive models useful in practice

Jung

Kashyap

Avati

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…17 However, to our knowledge, there have been limited published studies demonstrating translation of an automated discharge prediction model into clinical practice with analysis of impact on operational outcomes. 21 The objectives of this study were to develop and deploy a machine-learning-based discharge prediction tool and evaluate its impact on hospital patient flow.…”

Section: Introductionmentioning

confidence: 99%

Machine-learning-based hospital discharge predictions can support multidisciplinary rounds and decrease hospital length-of-stay

et al. 2020

View full text Add to dashboard Cite

BackgroundPatient flow directly affects quality of care, access and financial performance for hospitals. Multidisciplinary discharge-focused rounds have proven to minimise avoidable delays experienced by patients near discharge. The study objective was to support discharge-focused rounds by implementing a machine-learning-based discharge prediction model using real-time electronic health record (EHR) data. We aimed to evaluate model predictive performance and impact on hospital length-of-stay.MethodsDischarge prediction models were developed from hospitalised patients on four inpatient units between April 2016 and September 2018. Unit-specific models were implemented to make individual patient predictions viewable with the EHR patient track board. Predictive performance was measured prospectively for 12 470 patients (120 780 patient-predictions) across all units. A pre/poststudy design applying interrupted time series methods was used to assess the impact of the discharge prediction model on hospital length-of-stay.ResultsProspective discharge prediction performance ranged in area under the receiver operating characteristic curve from 0.70 to 0.80 for same-day and next-day predictions; sensitivity was between 0.63 and 0.83 and specificity between 0.48 and 0.80. Elapsed length-of-stay, counts of labs and medications, mobility assessments and measures of acute kidney injury were model features providing the most predictive value. Implementing the discharge predictions resulted in a reduction in hospital length-of-stay of over 12 hours on a medicine unit (p<0.001) and telemetry unit (p=0.002), while no changes were observed for the surgery unit (p=0.190) and second medicine unit (p<0.555).ConclusionsIncorporating automated patient discharge predictions into multidisciplinary rounds can support decreases in hospital length-of-stay. Variation in execution and impact across inpatient units existed.

show abstract

“…According to [8], there is no clinically validated system exists for real-time prediction of sepsis onset especially those that can work at bedside. According to seminal article published at BMJ by Seneviratne [9] recently, very few of the machine learning algorithms ever make it to the bedside; and even the most technology-literate academic medical centers are not routinely using AI in clinical workflows. Commonly, however, at the emergency (IUC) departments they use pen and paper to a large extent when recording emergency care procedures and measurements.…”

Section: Introductionmentioning

confidence: 99%