6562 Background: Patients with advanced cancer that utilize end of life planning see benefits including better quality of life and medical care that is more consistent with their values. We developed a 30-day mortality predictive model using a machine learning algorithm and integrated it into a clinical decision support system (CDSS) that encourages clinicians to use the serious illness conversation (SIC) guide—a standardized questionnaire and conversational tool that facilitates end-of-life planning. The CDSS was piloted in the thoracic oncology clinic. We evaluated clinicians’ use of this system and its impact on patient outcomes. Methods: Between 4/14/21-1/15/22, information about patients identified by the model was sent to clinical teams via the electronic health record (EHR) to assess eligibility for a SIC. We reviewed the EHR for patients identified, SIC completion, and level of agreement by oncologists with the model. We evaluated the SIC guide responses using descriptive statistics and assessed differences in rates of hospice referral, hospital visits, and 30-day mortality by SIC completion status. Chi-squared test was used for testing association. Results: 94 patients were evaluated for SIC eligibility. Of these, oncologists agreed with 48 (51%) model predictions and SIC was completed for 28 (58%) of those patients. A median of 2.5 SIC eligibility assessments per week were completed, with a median time of 4 days from prediction to assessment. Likewise, a median of 1 SIC per week was completed, with a median time of 20 days from SIC eligibility assessment to conversation. Regarding the responses to the SIC guide, out of 28 patients, 75% have an appropriate understanding of their illness; 64% want to be fully informed of their medical information while 21% prefer information to be limited. Common patient goals were “being comfortable” (54%), “being at home” (29%) and “being independent” (25%). The most prevalent patient fears were “family concerns” (29%) or “physical suffering” (25%). The clinician who performs the SIC most often recommended an “additional conversation with physician” (39%), “conversation with family” (36%), or “referral to palliative care” (18%). SIC completion was associated with an increased rate of enrollment in hospice (33% vs 14%, P= 0.03) on univariate analysis. SIC was not associated with a difference in 30-day mortality or hospital visits. Multivariable analysis is ongoing. Conclusions: The machine-learning powered CDSS was adopted by the oncology care team within a reasonable timeframe. However, even if an oncologist used and agreed with the CDSS, the rate of eventual completion of SIC was not 100%. Additional barriers to SIC will be studied to optimize the CDSS. SIC completion may lead to increased enrollment in hospice and should continue to be studied as a standard component of comprehensive cancer care.

No abstract

330 Background: Machine learning models are well-positioned to transform cancer care delivery by providing oncologists with more accurate or accessible information to augment clinical decisions. Many machine learning projects, however, focus on model accuracy without considering the impact of using the model in real-world settings and rarely carry forward to clinical implementation. We present a human-centered systems engineering approach to address clinical problems with workflow interventions utilizing machine learning algorithms. Methods: We aimed to develop a mortality predictive tool, using a Random Forest algorithm, to identify oncology patients at high risk of death within 30 days to move advance care planning (ACP) discussions earlier in the illness trajectory. First, a project sponsor defined the clinical need and requirements of an intervention. The data scientists developed the predictive algorithm using data available in the electronic health record (EHR). A multidisciplinary workgroup was assembled including oncology physicians, advanced practice providers, nurses, social workers, chaplain, clinical informaticists, and data scientists. Meeting bi-monthly, the group utilized human-centered design (HCD) methods to understand clinical workflows and identify points of intervention. The workgroup completed a workflow redesign workshop, a 90-minute facilitated group discussion, to integrate the model in a future state workflow. An EHR (Epic) analyst built the user interface to support the intervention per the group’s requirements. The workflow was piloted in thoracic oncology and bone marrow transplant with plans to scale to other cancer clinics. Results: Our predictive model performance on test data was acceptable (sensitivity 75%, specificity 75%, F-1 score 0.71, AUC 0.82). The workgroup identified a “quality of life coordinator” who: reviews an EHR report of patients scheduled in the upcoming 7 days who have a high risk of 30-day mortality; works with the oncology team to determine ACP clinical appropriateness; documents the need for ACP; identifies potential referrals to supportive oncology, social work, or chaplain; and coordinates the oncology appointment. The oncologist receives a reminder on the day of the patient’s scheduled visit. Conclusions: This workgroup is a viable approach that can be replicated at institutions to address clinical needs and realize the full potential of machine learning models in healthcare. The next steps for this project are to address end-user feedback from the pilot, expand the intervention to other cancer disease groups, and track clinical metrics.