Identification of Serious Illness Conversations in Unstructured Clinical Notes Using Deep Neural Networks

Chien, Isabel; Shi, Alvin; Chan, Alex J.

doi:10.1007/978-3-030-12738-1_15

Cited by 3 publications

(6 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Third, because our sample was limited to clinical notes from a single tertiary care hospital in northeastern United States and lacked racial diversity, our algorithm may not be generalizable to other hospitals, ICU populations, or geographic areas. Fourth, as noted in other studies [ 34 , 44 , 52 ], our methods were dependent on the quantity and quality of documentation that exist in the EHR, so it is possible that some family-related documentation or actual interaction with and involvement of families may have been missed. Moreover, our models may not fully account for all possible confounders, and we were unable to capture other factors that may impact the relationship between family involvement and patient outcomes.…”

Section: Discussionmentioning

confidence: 99%

Associations Between Family Member Involvement and Outcomes of Patients Admitted to the Intensive Care Unit: Retrospective Cohort Study

Gray¹,

Kwok²,

Khuyen³

et al. 2022

JMIR Med Inform

Self Cite

View full text Add to dashboard Cite

Background Little is known about family member involvement, by relationship status, for patients treated in the intensive care unit (ICU). Objective Using documentation of family interactions in clinical notes, we examined associations between child and spousal involvement and ICU patient outcomes, including goals of care conversations (GOCCs), limitations in life-sustaining therapy (LLST), and 3-month mortality. Methods Using a retrospective cohort design, the study included a total of 858 adult patients treated between 2008 and 2012 in the medical ICU at a tertiary care center in northeastern United States. Clinical notes generated within the first 48 hours of admission to the ICU were used with standard machine learning methods to predict patient outcomes. We used natural language processing methods to identify family-related documentation and abstracted sociodemographic and clinical characteristics of the patients from the medical record. Results Most of the 858 patients were White (n=650, 75.8%); 437 (50.9%) were male, 479 (55.8%) were married, and the median age was 68.4 (IQR 56.5-79.4) years. Most patients had documented GOCC (n=651, 75.9%). In adjusted regression analyses, child involvement (odds ratio [OR] 0.81; 95% CI 0.49-1.34; P=.41) and child plus spouse involvement (OR 1.28; 95% CI 0.8-2.03; P=.3) were not associated with GOCCs compared to spouse involvement. Child involvement was not associated with LLST when compared to spouse involvement (OR 1.49; 95% CI 0.89-2.52; P=.13). However, child plus spouse involvement was associated with LLST (OR 1.6; 95% CI 1.02-2.52; P=.04). Compared to spouse involvement, there were no significant differences in the 3-month mortality by family member type, including child plus spouse involvement (OR 1.38; 95% CI 0.91-2.09; P=.13) and child involvement (OR 1.47; 95% CI 0.9-2.41; P=.12). Conclusions Our findings demonstrate that statistical models derived from text analysis in the first 48 hours of ICU admission can predict patient outcomes. Early child plus spouse involvement was associated with LLST, suggesting that decisions about LLST were more likely to occur when the child and spouse were both involved compared to the involvement of only the spouse. More research is needed to further understand the involvement of different family members in ICU care and its association with patient outcomes.

show abstract

Section: Discussionmentioning

confidence: 99%

Associations Between Family Member Involvement and Outcomes of Patients Admitted to the Intensive Care Unit: Retrospective Cohort Study

Gray¹,

Kwok²,

Khuyen³

et al. 2022

JMIR Med Inform

Self Cite

View full text Add to dashboard Cite

show abstract

“…[8][9][10] In using NLP for clinical research, it is important to implement robust approaches to address NLP-related misclassification in study design and analysis. 9,10 Researchers in the fields of palliative care and serious illness communication have shown interest in using NLP 7,[11][12][13][14] to measure the occurrence and timing of EHR-documented goals-of-care discussions, an outcome that reflects clinicians' assessment and documentation of patients' values, goals, and treatment preferences. 15,16 This outcome represents a guideline-recommended best practice, [16][17][18][19][20] an area of ongoing deficiencies, [21][22][23] and a mediator of delivery of patient-centered care.…”

Section: Introductionmentioning

confidence: 99%

“…16,[24][25][26] However, goalsof-care discussions are difficult to measure from structured EHR data or claims data, and their rarity within free-text records makes them costly to manually abstract at scale. 13,[27][28][29][30] Although NLP models have been developed to measure this and related constructs, 6,7,[12][13][14]31 the linguistic complexity encountered in documented goals-of-care discussions continues to challenge NLP, and there is ongoing interest in refining NLP approaches to improve performance. 7,14 In this study, we used deep-learning NLP to measure the primary outcome of EHR-documented goals-of-care discussions in a large pragmatic trial of a communication-priming intervention for hospitalized patients.…”

Section: Introductionmentioning

confidence: 99%

“…[1][2][3] However, NLP can introduce inaccuracies and misclassify outcomes, particularly when measuring complex constructs. [4][5][6][7] Many statistical procedures common in clinical research ignore misclassification, and applying these procedures to imperfectly measured outcomes can lead to underpowered studies and improper estimates. [8][9][10] In using NLP for clinical research, it is important to implement robust approaches to address NLP-related misclassification in study design and analysis.…”

Section: Introductionmentioning

confidence: 99%

“…13,[27][28][29][30] Although NLP models have been developed to measure this and related constructs, 6,7,[12][13][14]31 the linguistic complexity encountered in documented goals-of-care discussions continues to challenge NLP, and there is ongoing interest in refining NLP approaches to improve performance. 7,14 In this study, we used deep-learning NLP to measure the primary outcome of EHR-documented goals-of-care discussions in a large pragmatic trial of a communication-priming intervention for hospitalized patients. 32,33 We evaluated the sensitivity, specificity, and predictive values of our deeplearning NLP model through receiver operating characteristic (ROC) curves and precision-recall analyses and examined the effect of NLP-related misclassification on study power.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Assessment of Natural Language Processing of Electronic Health Records to Measure Goals-of-Care Discussions as a Clinical Trial Outcome

et al. 2023

View full text Add to dashboard Cite

ImportanceMany clinical trial outcomes are documented in free-text electronic health records (EHRs), making manual data collection costly and infeasible at scale. Natural language processing (NLP) is a promising approach for measuring such outcomes efficiently, but ignoring NLP-related misclassification may lead to underpowered studies.ObjectiveTo evaluate the performance, feasibility, and power implications of using NLP to measure the primary outcome of EHR-documented goals-of-care discussions in a pragmatic randomized clinical trial of a communication intervention.Design, Setting, and ParticipantsThis diagnostic study compared the performance, feasibility, and power implications of measuring EHR-documented goals-of-care discussions using 3 approaches: (1) deep-learning NLP, (2) NLP-screened human abstraction (manual verification of NLP-positive records), and (3) conventional manual abstraction. The study included hospitalized patients aged 55 years or older with serious illness enrolled between April 23, 2020, and March 26, 2021, in a pragmatic randomized clinical trial of a communication intervention in a multihospital US academic health system.Main Outcomes and MeasuresMain outcomes were natural language processing performance characteristics, human abstractor-hours, and misclassification-adjusted statistical power of methods of measuring clinician-documented goals-of-care discussions. Performance of NLP was evaluated with receiver operating characteristic (ROC) curves and precision-recall (PR) analyses and examined the effects of misclassification on power using mathematical substitution and Monte Carlo simulation.ResultsA total of 2512 trial participants (mean [SD] age, 71.7 [10.8] years; 1456 [58%] female) amassed 44 324 clinical notes during 30-day follow-up. In a validation sample of 159 participants, deep-learning NLP trained on a separate training data set from identified patients with documented goals-of-care discussions with moderate accuracy (maximal F1 score, 0.82; area under the ROC curve, 0.924; area under the PR curve, 0.879). Manual abstraction of the outcome from the trial data set would require an estimated 2000 abstractor-hours and would power the trial to detect a risk difference of 5.4% (assuming 33.5% control-arm prevalence, 80% power, and 2-sided α = .05). Measuring the outcome by NLP alone would power the trial to detect a risk difference of 7.6%. Measuring the outcome by NLP-screened human abstraction would require 34.3 abstractor-hours to achieve estimated sensitivity of 92.6% and would power the trial to detect a risk difference of 5.7%. Monte Carlo simulations corroborated misclassification-adjusted power calculations.Conclusions and RelevanceIn this diagnostic study, deep-learning NLP and NLP-screened human abstraction had favorable characteristics for measuring an EHR outcome at scale. Adjusted power calculations accurately quantified power loss from NLP-related misclassification, suggesting that incorporation of this approach into the design of studies using NLP would be beneficial.

show abstract

Detecting Goals of Care Conversations in Clinical Notes with Active Learning

Weissenbacher,

Courtright,

Rawal

et al. 2024

Preprint

View full text Add to dashboard Cite

ObjectiveGoals Of Care (GOC) discussions are an increasingly used quality metric in serious illness care and research. Wide variation in documentation practices within the Electronic Health Record (EHR) presents challenges for reliable measurement of GOC discussions. Novel natural language processing approaches are needed to capture GOC discussions documented in real-world samples of seriously ill hospitalized patients’ EHR notes, a corpus with a very low event prevalence.MethodsTo automatically detect utterances documenting GOC discussions outside of dedicated GOC note types, we proposed an ensemble of classifiers aggregating the predictions of rule-based, feature-based, and three transformers-based classifiers. We trained our classifier on 600 manually annotated EHR notes among patients with serious illnesses. Our corpus exhibited an extremely imbalanced ratio between utterances discussing GOC and utterances that do not. This ratio challenges standard supervision methods to train a classifier. Therefore, we trained our classifier with active learning.ResultsUsing active learning, we reduced the annotation cost to fine-tune our ensemble by 70% while improving its performance in our test set of 176 EHR notes, with 0.557 F1-score for utterance classification and 0.629 for note classification.ConclusionWhen classifying notes, with a true positive rate of 72% (13/18) and false positive rate of 8% (13/158), our performance may be sufficient for deploying our classifier in the EHR to facilitate point-of-care access to GOC conversations documented outside of dedicated notes types, without overburdening clinicians with false positives. Improvements are needed before using it to enrich trial populations or as an outcome measure.

show abstract

Identification of Serious Illness Conversations in Unstructured Clinical Notes Using Deep Neural Networks

Cited by 3 publications

References 25 publications

Associations Between Family Member Involvement and Outcomes of Patients Admitted to the Intensive Care Unit: Retrospective Cohort Study

Associations Between Family Member Involvement and Outcomes of Patients Admitted to the Intensive Care Unit: Retrospective Cohort Study

Assessment of Natural Language Processing of Electronic Health Records to Measure Goals-of-Care Discussions as a Clinical Trial Outcome

Detecting Goals of Care Conversations in Clinical Notes with Active Learning

Contact Info

Product

Resources

About