Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation

Huang, Kexin; Singh, Abhishek; Chen, Sitong; Moseley, Edward T.; Deng, Chih-Ying; George, Naomi

doi:10.18653/v1/2020.clinicalnlp-1.11

Cited by 47 publications

(32 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although they did not report any evidence of the superiority of the language model based approach, our works still complement each other. Another complementary worth mentioning work is Huang et al ( 7 ), which tried to overcome transformer input size limit for 30-day re-admission task on the same dataset, but unlike us, pursued simple hueristic rather than deep-learning solutions and reported smaller improvements. Unfortunately, we were not able to use their pre-trained language model and are not aware of any follow up study that did.…”

mentioning

confidence: 89%

Predicting Clinical Events Based on Raw Text: From Bag-of-Words to Attention-Based Transformers

Roussinov¹,

Conkie²,

Patterson³

et al. 2022

Front. Digit. Health

View full text Add to dashboard Cite

Identifying which patients are at higher risks of dying or being re-admitted often happens to be resource- and life- saving, thus is a very important and challenging task for healthcare text analytics. While many successful approaches exist to predict such clinical events based on categorical and numerical variables, a large amount of health records exists in the format of raw text such as clinical notes or discharge summaries. However, the text-analytics models applied to free-form natural language found in those notes are lagging behind the break-throughs happening in the other domains and remain to be primarily based on older bag-of-words technologies. As a result, they rarely reach the accuracy level acceptable for the clinicians. In spite of their success in other domains, the superiority of deep neural approaches over classical bags of words for this task has not yet been convincingly demonstrated. Also, while some successful experiments have been reported, the most recent break-throughs due to the pre-trained language models have not yet made their ways into the medical domain. Using a publicly available healthcare dataset, we have explored several classification models to predict patients' re-admission or a fatality based on their discharge summaries and established that 1) The performance of the neural models used in our experiments convincingly exceeds those based on bag-of-words by several percentage points as measured by the standard metrics. 2) This allows us to achieve the accuracy typically acceptable by the clinicians as of practical use (area under the ROC curve above 0.70) for the majority of our prediction targets. 3) While the pre-trained attention-based transformer performed only on par with the model that averages word embeddings when applied to full length discharge summaries, the transformer still handles shorter text segments substantially better, at times with the margin of 0.04 in the area under the ROC curve. Thus, our findings extend the success of pre-trained language models reported in other domains to the task of clinical event prediction, and likely to other text-classification tasks in the healthcare analytics domain. 4) We suggest several models to overcome the transformers' major drawback (their input size limitation), and confirm that this is crucial to achieve their top performance. Our modifications are domain agnostic, and thus can be applied in other applications where the text inputs exceed 200 words. 5) We have successfully demonstrated how non-text attributes (such as patient age, demographics, type of admission etc.) can be combined with text to gain additional improvements for several prediction targets. We include extensive ablation studies showing the impact of the training size, and highlighting the tradeoffs between the performance and the resources needed.

show abstract

mentioning

confidence: 89%

Predicting Clinical Events Based on Raw Text: From Bag-of-Words to Attention-Based Transformers

Roussinov¹,

Conkie²,

Patterson³

et al. 2022

Front. Digit. Health

View full text Add to dashboard Cite

show abstract

“…[8][9][10] Recently, studies have reported that a new deep learning-based architecture, named "transformers," achieved state-of-the-art performance for a number of benchmark tasks [11][12][13][14][15][16] in the general English domain. Although several studies have examined transformer-based models for clinical individually, [17][18][19][20][21] there is no study that has systematically explored and compared their performance in the biomedical domain. In addition, there is a lack of package with pretrained clinical transformers that could facilitate researchers and other users adopting these state-of-the-art NLP models in various downstream clinical NLP tasks.…”

Section: Introductionmentioning

confidence: 99%

Clinical concept extraction using transformers

Yang

Bian

Hogan

et al. 2020

Journal of the American Medical Informatics Association

110

View full text Add to dashboard Cite

Objective The goal of this study is to explore transformer-based models (eg, Bidirectional Encoder Representations from Transformers [BERT]) for clinical concept extraction and develop an open-source package with pretrained clinical models to facilitate concept extraction and other downstream natural language processing (NLP) tasks in the medical domain. Methods We systematically explored 4 widely used transformer-based architectures, including BERT, RoBERTa, ALBERT, and ELECTRA, for extracting various types of clinical concepts using 3 public datasets from the 2010 and 2012 i2b2 challenges and the 2018 n2c2 challenge. We examined general transformer models pretrained using general English corpora as well as clinical transformer models pretrained using a clinical corpus and compared them with a long short-term memory conditional random fields (LSTM-CRFs) mode as a baseline. Furthermore, we integrated the 4 clinical transformer-based models into an open-source package. Results and Conclusion The RoBERTa-MIMIC model achieved state-of-the-art performance on 3 public clinical concept extraction datasets with F1-scores of 0.8994, 0.8053, and 0.8907, respectively. Compared to the baseline LSTM-CRFs model, RoBERTa-MIMIC remarkably improved the F1-score by approximately 4% and 6% on the 2010 and 2012 i2b2 datasets. This study demonstrated the efficiency of transformer-based models for clinical concept extraction. Our methods and systems can be applied to other clinical tasks. The clinical transformer package with 4 pretrained clinical models is publicly available at https://github.com/uf-hobi-informatics-lab/ClinicalTransformerNER. We believe this package will improve current practice on clinical concept extraction and other tasks in the medical domain.

show abstract

“…If it indeed reduces the time and memory consumed by the experiment and improves the recommendation performance, then, we will improve it within the existing method. In addition, BERT, which involves knowledge in the medical field, such as Clin-icalBERT [34] and BioBERT [35], will also serve as our follow-up research work.…”

Section: Discussionmentioning

confidence: 99%

Deep Personalized Medical Recommendations Based on the Integration of Rating Features and Review Sentiment Analysis

Zhou

et al. 2021

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

To comply with the rapid development of big data in mobile services, an increasing number of websites have begun to provide users with recommendation decisions in various areas, like shopping, tourism, food, and medical treatment. However, there are still some challenges in the field of medical recommendation systems, such as the lack of personalized medical recommendations and the problem of data sparseness, which seriously restricts the effectiveness of such recommendations. In this paper, we propose a personalized medical recommendation method based on a convolutional neural network that integrates revised ratings and review text, called revised rating and review based on a convolutional neural network (RR&R-CNN). First, the review text is divided into user and doctor datasets, and BERT vectorized representations are performed on them. Moreover, the original rating features are revised by adding the sentiment analysis values of the review text. Then, the vectorized review text and the revised rating features are spliced together and input into the convolutional neural network to extract the deep nonlinear feature vectors of both users and doctors. Finally, we use a factorization machine for feature interaction. We conduct comparison experiments based on a Yelp dataset in the “Health & Medical” category. The experimental results confirm the conclusion that RR&R-CNN has a better effect compared to a traditional method.

show abstract

Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation

Cited by 47 publications

References 23 publications

Predicting Clinical Events Based on Raw Text: From Bag-of-Words to Attention-Based Transformers

Predicting Clinical Events Based on Raw Text: From Bag-of-Words to Attention-Based Transformers

Clinical concept extraction using transformers

Deep Personalized Medical Recommendations Based on the Integration of Rating Features and Review Sentiment Analysis

Contact Info

Product

Resources

About