A validation of machine learning-based risk scores in the prehospital setting

Abstract

Background: The triage of patients in pre-hospital care is a difficult task, and improved risk assessment tools are needed both at the dispatch center and on the ambulance to differentiate between low- and high-risk patients. This study develops and validates a machine learning-based approach to predicting hospital outcomes based on routinely collected prehospital data. Methods: Dispatch, ambulance, and hospital data were collected in one Swedish region from 2016 to 2017. Dispatch center and ambulance records were used to develop gradient boosting models predicting hospital admission, critical care (defined as admission to an intensive care unit or in-hospital mortality), and two-day mortality. Model predictions were used to generate composite risk scores which were compared to National Early Warning System (NEWS) scores and actual dispatched priorities in a similar but prospectively gathered dataset from 2018. Results: A total of 38203 patients were included from 2016-2018. Concordance indexes (or area under the receiver operating characteristics curve) for dispatched priorities ranged from 0.51 to 0.66, while those for NEWS scores ranged from 0.66 to 0.85. Concordance ranged from 0.71 to 0.80 for risk scores based only on dispatch data, and 0.79 to 0.89 for risk scores including ambulance data. Dispatch data-based risk scores consistently outperformed dispatched priorities in predicting hospital outcomes, while models including ambulance data also consistently outperformed NEWS scores. Model performance in the prospective test dataset was similar to that found using cross-validation, and calibration was comparable to that of NEWS scores.

Registered nurses' understandings of emergency medical dispatch center work: A qualitative phenomenographic interview study

“…Furthermore, lack of clarity in patient complaints often poses great challenges for dispatchers and RNs. To aid assessments, the use of computerized clinical decision support systems (CDSS) is common (Holmström et al, 2020; Spangler et al, 2019). EMDC work may thus be conceived of as either simple compliance with a set of pre‐defined guidelines (Booker et al, 2018), or as requiring more advanced competence (Rutenberg & Greenberg, 2012).…”

Can Prehospital Data Improve Early Identification of Sepsis in Emergency Department? An Integrative Review of Machine Learning Approaches

“…Limited prehospital data have been used in developing ED prediction models to improve ED outcomes. Of the 12 studies reviewed that used prehospital data for ED decision-making; 1 study improved ED operations by forecasting number of arrivals to reduce overcrowding in ED 14 ; 4 studies predicted patient outcomes such as in-hospital mortality, survival rate, and return of spontaneous circulation (ROSC) 15 16 17 18 ; 2 studies identified specific risk or early warning scores for patient outcomes such as higher acuity or short-term in-hospital mortality 19 20 ; 2 studies made decisions in the field prior to arrival to ED such as triage patient disposition to specialized centers with appropriate medical capabilities (e.g., trauma centers or aortic surgery centers) 21 22 ; and 3 studies identified time-sensitive conditions. 21 22 23 All studies reported improvement in ED operations or ED outcomes as described in Table 1 .…”

“…Of the nine studies included in the review, common prehospital data elements used in ML models targeting early detection of time-sensitive conditions or prediction of patient outcomes include the following: (1) demographics, such as age and gender; chief complaints 15 16 20 21 22 24 25 ; (2) prehospital level of consciousness such as Glasgow coma scale (GCS) 19 20 22 25 ; (3) prehospital vital signs recorded in the ambulance or in the field 16 21 22 ; (4) prehospital actions, characterized as intubation, cardiopulmonary resuscitation (CPR), and chest decompression 16 18 21 22 ; or (5) prehospital situational assessments, described as mechanism of injury, witnessed arrest or CPR initiation, and presence of speech deficit and facial and limb weakness prior to arrival to ED. 15 18 21 22 Table 1 summarizes studies using prehospital data for clinical decision-making in ED based on their targets and modeling approaches.…”

Development of machine learning model for predicting hospitalization in the prehospital setting

“…3 Another prognostic study using machine learning reported that a model using ≥1000 predictors has good discrimination ability. 5 However, these models, including machine learning models, generally require many predictors to achieve high prediction ability-this becomes a bar to implement prediction models in the real clinical setting.…”

“…2 To address this concern, studies have developed models for predicting patient outcomes for successful risk stratification in the prehospital setting. [2][3][4][5][6] For example, a multicenter study of 4950 trauma patients developed a model to predict severe injury defined as an Injury Severity Score greater than 15, and the model had a high discrimination ability (C statistic of 0.823). 3 Another prognostic study using machine learning reported that a model using ≥1000 predictors has good discrimination ability.…”