ObjectivesTo develop and validate a model for prediction of near-term in-hospital mortality among patients with COVID-19 by application of a machine learning (ML) algorithm on time-series inpatient data from electronic health records.MethodsA cohort comprised of 567 patients with COVID-19 at a large acute care healthcare system between 10 February 2020 and 7 April 2020 observed until either death or discharge. Random forest (RF) model was developed on randomly drawn 70% of the cohort (training set) and its performance was evaluated on the rest of 30% (the test set). The outcome variable was in-hospital mortality within 20–84 hours from the time of prediction. Input features included patients’ vital signs, laboratory data and ECG results.ResultsPatients had a median age of 60.2 years (IQR 26.2 years); 54.1% were men. In-hospital mortality rate was 17.0% and overall median time to death was 6.5 days (range 1.3–23.0 days). In the test set, the RF classifier yielded a sensitivity of 87.8% (95% CI: 78.2% to 94.3%), specificity of 60.6% (95% CI: 55.2% to 65.8%), accuracy of 65.5% (95% CI: 60.7% to 70.0%), area under the receiver operating characteristic curve of 85.5% (95% CI: 80.8% to 90.2%) and area under the precision recall curve of 64.4% (95% CI: 53.5% to 75.3%).ConclusionsOur ML-based approach can be used to analyse electronic health record data and reliably predict near-term mortality prediction. Using such a model in hospitals could help improve care, thereby better aligning clinical decisions with prognosis in critically ill patients with COVID-19.

6562 Background: Patients with advanced cancer that utilize end of life planning see benefits including better quality of life and medical care that is more consistent with their values. We developed a 30-day mortality predictive model using a machine learning algorithm and integrated it into a clinical decision support system (CDSS) that encourages clinicians to use the serious illness conversation (SIC) guide—a standardized questionnaire and conversational tool that facilitates end-of-life planning. The CDSS was piloted in the thoracic oncology clinic. We evaluated clinicians’ use of this system and its impact on patient outcomes. Methods: Between 4/14/21-1/15/22, information about patients identified by the model was sent to clinical teams via the electronic health record (EHR) to assess eligibility for a SIC. We reviewed the EHR for patients identified, SIC completion, and level of agreement by oncologists with the model. We evaluated the SIC guide responses using descriptive statistics and assessed differences in rates of hospice referral, hospital visits, and 30-day mortality by SIC completion status. Chi-squared test was used for testing association. Results: 94 patients were evaluated for SIC eligibility. Of these, oncologists agreed with 48 (51%) model predictions and SIC was completed for 28 (58%) of those patients. A median of 2.5 SIC eligibility assessments per week were completed, with a median time of 4 days from prediction to assessment. Likewise, a median of 1 SIC per week was completed, with a median time of 20 days from SIC eligibility assessment to conversation. Regarding the responses to the SIC guide, out of 28 patients, 75% have an appropriate understanding of their illness; 64% want to be fully informed of their medical information while 21% prefer information to be limited. Common patient goals were “being comfortable” (54%), “being at home” (29%) and “being independent” (25%). The most prevalent patient fears were “family concerns” (29%) or “physical suffering” (25%). The clinician who performs the SIC most often recommended an “additional conversation with physician” (39%), “conversation with family” (36%), or “referral to palliative care” (18%). SIC completion was associated with an increased rate of enrollment in hospice (33% vs 14%, P= 0.03) on univariate analysis. SIC was not associated with a difference in 30-day mortality or hospital visits. Multivariable analysis is ongoing. Conclusions: The machine-learning powered CDSS was adopted by the oncology care team within a reasonable timeframe. However, even if an oncologist used and agreed with the CDSS, the rate of eventual completion of SIC was not 100%. Additional barriers to SIC will be studied to optimize the CDSS. SIC completion may lead to increased enrollment in hospice and should continue to be studied as a standard component of comprehensive cancer care.

330 Background: Machine learning models are well-positioned to transform cancer care delivery by providing oncologists with more accurate or accessible information to augment clinical decisions. Many machine learning projects, however, focus on model accuracy without considering the impact of using the model in real-world settings and rarely carry forward to clinical implementation. We present a human-centered systems engineering approach to address clinical problems with workflow interventions utilizing machine learning algorithms. Methods: We aimed to develop a mortality predictive tool, using a Random Forest algorithm, to identify oncology patients at high risk of death within 30 days to move advance care planning (ACP) discussions earlier in the illness trajectory. First, a project sponsor defined the clinical need and requirements of an intervention. The data scientists developed the predictive algorithm using data available in the electronic health record (EHR). A multidisciplinary workgroup was assembled including oncology physicians, advanced practice providers, nurses, social workers, chaplain, clinical informaticists, and data scientists. Meeting bi-monthly, the group utilized human-centered design (HCD) methods to understand clinical workflows and identify points of intervention. The workgroup completed a workflow redesign workshop, a 90-minute facilitated group discussion, to integrate the model in a future state workflow. An EHR (Epic) analyst built the user interface to support the intervention per the group’s requirements. The workflow was piloted in thoracic oncology and bone marrow transplant with plans to scale to other cancer clinics. Results: Our predictive model performance on test data was acceptable (sensitivity 75%, specificity 75%, F-1 score 0.71, AUC 0.82). The workgroup identified a “quality of life coordinator” who: reviews an EHR report of patients scheduled in the upcoming 7 days who have a high risk of 30-day mortality; works with the oncology team to determine ACP clinical appropriateness; documents the need for ACP; identifies potential referrals to supportive oncology, social work, or chaplain; and coordinates the oncology appointment. The oncologist receives a reminder on the day of the patient’s scheduled visit. Conclusions: This workgroup is a viable approach that can be replicated at institutions to address clinical needs and realize the full potential of machine learning models in healthcare. The next steps for this project are to address end-user feedback from the pilot, expand the intervention to other cancer disease groups, and track clinical metrics.

No abstract

433 Background: We previously reported the implementation of a machine learning (ML) model for mortality prediction that was integrated into a CDSS encouraging clinicians to have a SIC with at-risk cancer patients. The clinical utility of a ML model can change after implementation due to fluctuations in the organization’s patient population and clinical practices. It is important to establish a workflow to monitor and continually reinforce ML-powered CDSS to ensure that it continues to benefit patients. We report a workgroup structure that incorporates data driven evaluation of ML model performance and feedback from CDSS end users to optimize the acceptability of the CDSS. Methods: The workflow was piloted in the gastrointestinal (GI) oncology clinic from 11/2021-5/2022. A workgroup including members of the implementation team and end-users of the CDSS met monthly to review 1) a dashboard that displays model performance, 2) an electronic health record (EHR) report that summarizes use of the CDSS, 3) feedback from end users regarding their opinion of the CDSS and any barriers to implementation. We evaluated the accuracy of model predictions among subgroups as defined by mortality and unplanned hospital admissions or ED visit rates. Fisher’s Exact Test was used to identify differences between categorical variables. Numeric values including incidence rate ratios (IRRs) adjusted for age, sex, race, and gender with 95% confidence intervals (CIs) were calculated using Poisson regression. Results: 119 patients were evaluated by the model and 50 (42%) were assessed as high-risk. In the high-risk group, the oncology team evaluated 39 (78%) patients for appropriateness of a SIC; SIC was completed with 5 (10%) patients. During workgroup meetings, physicians shared that some of the high-risk predictions were for patients undergoing curative intent therapy. 0 out of 24 patients who received curative treatment died and 5 out of 26 patients who receive palliative treatment died. The log-rank p-value of 0.03 indicates that the survival distribution differs significantly over time between two groups. The adjusted IRR for unplanned hospital visits (palliative vs curative) was 2.55 (1.3-5.0). Adjusted mean hospital visits per month were 0.34 (0.21-0.51) vs 0.13 (0.06-0.21). Conclusions: The workgroup format is a feasible method to continuously review acceptability of a ML-powered CDSS. It may evaluate critical feedback from end users in a holistic manner that can augment a data driven evaluation of the model performance. The data implies that patients undergoing curative therapy have a decreased risk for mortality and unplanned hospital admissions or ED visits. The CDSS may be optimized by excluding these patients; however, longer follow up of this sub-population is needed to confirm that they have no additional risk factors.