Adapting Bidirectional Encoder Representations from Transformers (BERT) to Assess Clinical Semantic Textual Similarity: Algorithm Development and Validation Study

Abstract

Background Natural Language Understanding enables automatic extraction of relevant information from clinical text data, which are acquired every day in hospitals. In 2018, the language model Bidirectional Encoder Representations from Transformers (BERT) was introduced, generating new state-of-the-art results on several downstream tasks. The National NLP Clinical Challenges (n2c2) is an initiative that strives to tackle such downstream tasks on domain-specific clinical data. In this paper, we present the results of our participation in the 2019 n2c2 and related work completed thereafter. Objective The objective of this study was to optimally leverage BERT for the task of assessing the semantic textual similarity of clinical text data. Methods We used BERT as an initial baseline and analyzed the results, which we used as a starting point to develop 3 different approaches where we (1) added additional, handcrafted sentence similarity features to the classifier token of BERT and combined the results with more features in multiple regression estimators, (2) incorporated a built-in ensembling method, M-Heads, into BERT by duplicating the regression head and applying an adapted training strategy to facilitate the focus of the heads on different input patterns of the medical sentences, and (3) developed a graph-based similarity approach for medications, which allows extrapolating similarities across known entities from the training set. The approaches were evaluated with the Pearson correlation coefficient between the predicted scores and ground truth of the official training and test dataset. Results We improved the performance of BERT on the test dataset from a Pearson correlation coefficient of 0.859 to 0.883 using a combination of the M-Heads method and the graph-based similarity approach. We also show differences between the test and training dataset and how the two datasets influenced the results. Conclusions We found that using a graph-based similarity approach has the potential to extrapolate domain specific knowledge to unseen sentences. We observed that it is easily possible to obtain deceptive results from the test dataset, especially when the distribution of the data samples is different between training and test datasets.

Impact of a Clinical Text–Based Fall Prediction Model on Preventing Extended Hospital Stays for Elderly Inpatients: Model Development and Performance Evaluation

“…It has often been reported that BERT exhibits high performance, even with clinical text [ 36 - 39 ]. This is also true for this study, in which a model combining BERT and Bi-LSTM using clinical text recorded in daily practice allowed for fall prediction with an accuracy equal to or higher than that of conventional risk assessment tools.…”

Graph-Augmented Cyclic Learning Framework for Similarity Estimation of Medical Clinical Notes

“…Therefore, it is suitable for evaluating the relationship between two sentences such as similarity. However, previous research [9] showed that Bioclinical BERT cannot handle well with prescriptions where less semantic information can be reflected. The author proposed a medication graph to enhance the backbone BERT model, but the graph is constructed from local data and cannot indicate relationships of unseen drug pairs objectively.…”

“…End-to-end deep learning-based models with fine tuning are the mainstream in this domain. State-of-the-art models like BERT [8] and its variants, and XLNet [18], are modified and involved in similarity calculating with task-specific fine tuning [9], [19]. Also, multitask learning (MTL) paradigm is widely used and verified to learn powerful representations from multiple data [20].…”

“…Only a professional doctor can distinguish them by potential diseases, prognosis, and drug relationships. Deep learning-based models like BERT [8] that are trained on semantically rich texts achieve huge success on general NLP tasks, but they often produce a sub-optimal performance for the MedSTS task [9]. Also, these methods often require a large amount of training data, whereas in the clinical domain there are always limited training data due to the sensitive patient information and time-consuming annotation.…”

Predicting Postoperative Mortality With Deep Neural Networks and Natural Language Processing: Model Development and Validation

“…Bidirectional encoder representations from transformers (BERT) is a contextualized embedding method that preserves the distance of meanings with multihead attention [ 17 ]. After pretrained on the relevant corpora and proper architecture modification, BERT extracts meaningful embeddings from clinical text [ 18 , 19 ].…”