MT-clinical BERT: scaling clinical information extraction with multitask learning

Mulyar, Andriy; Uzuner, Özlem; McInnes, Bridget T.

doi:10.1093/jamia/ocab126

Cited by 29 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, there have been various attempts to develop domain-specific LMs to extract key information from the respective custom text data. Similarly, clinicalBERT is trained on 2 million clinical notes in the MIMIC-III v1.4 database; – BioBERT is a biomedical-specific language model; mBERT is developed for multilingual machine translation tasks; patentBERT is built for patent classification; and FinBERT is built for financial NLP tasks . Some of the recently built LMs that have specific applications to process engineering include Ex-SciBERT, which deals with hydrogen production from alcohol-related literature for sentence classification, and NER; process-BERT was built for process informatics; and Battery-BERT was developed on battery-related scientific articles.…”

Section: Background and Contributionmentioning

confidence: 99%

Recycle-BERT: Extracting Knowledge about Plastic Waste Recycling by Natural Language Processing

Kumar,

Bakshi,

Ramteke

et al. 2023

ACS Sustainable Chem. Eng.

View full text Add to dashboard Cite

Managing waste plastic is a serious global challenge since most of this waste is either landfilled, incinerated, burned in the open, or littered. Each of these approaches has a large environmental impact. Establishing a circular economy of plastics requires its recovery and recycling, and much effort is now focused in this direction. The body of literature on approaches for managing the end of life of plastics is growing exponentially, making it increasingly difficult to segregate the most relevant information across multiple articles. Such work is extremely timeand effort-consuming, particularly when performed manually. To address this issue, in this study, we propose a methodology based on natural language processing (NLP) for automatically extracting and compiling information that is most relevant to a selected category of plastics. In the developed methodology, the research articles are first extracted with the help of a science-direct Elsevier Application Programming Interface key by utilizing a set of keywords such as "polyethylene recycle methods", "polyethylene terephthalate recycle methods", "polypropylene recycle methods", and "polystyrene recycle methods" for relevant articles. Extracted articles are processed to address two fundamental problems; (i) classification and (ii) question and answer (Q&A) related to literature pertaining to plastic waste recycling. To this extent, we developed a bundle of NLP tools called Recycle-Bidirectional Encoder Representations from Transformers (BERT). Under the hood, Recycle-BERT comprised five language models, (1) Class-BERT, for classifying the literature as relevant or nonrelevant; (2) Catalyst-BERT, for extracting catalyst details for recycling; (3) Method-BERT, for finding the methods enlisted in the literature for recycling; (4) Reactant-BERT to identify the reactants used for waste recycling; and (5) Product-BERT to pinpoint products obtained from recycling. We have evaluated the performance of the developed models based on the metrics such as accuracy and F1-score. For the classification task, an accuracy metric value of 0.974 is obtained for the test data set. Similarly, the metric F1-score values for the Q&A task are 0.7646, 0.8014, 0.8221, and 0.8512 for the test data set for Catalyst-BERT, Method-BERT, Reactant-BERT, and Product-BERT, respectively. The results indicate the proposed NLP-based model's ability to extract essential information from the literature related to plastic waste processing, aiding suitable recommendations to assist transformation to a sustainable circular economy.

show abstract

Section: Background and Contributionmentioning

confidence: 99%

Recycle-BERT: Extracting Knowledge about Plastic Waste Recycling by Natural Language Processing

Kumar,

Bakshi,

Ramteke

et al. 2023

ACS Sustainable Chem. Eng.

View full text Add to dashboard Cite

show abstract

“…In the clinical domain, Biseda et al [33] predict ICD codes from hospital notes by utilizing the Clinical BERT variant. Mulyar et al [34] developed Multitask-Clinical BERT for multitasking information extraction.…”

Section: Previous Workmentioning

confidence: 99%

Weighted Co-Occurrence Bio-Term Graph for Unsupervised Word Sense Disambiguation in the Biomedical Domain

et al. 2023

View full text Add to dashboard Cite

Word Sense Disambiguation (WSD) is a significant and challenging task for text understanding and processing. This paper presents an unsupervised approach based on Weighted Co-occurrence bio-Term Graph (WCOTG) for performing WSD in the biomedical domain. The graph is automatically created from biomedical terms that are extracted from a corpus of downloaded scientific abstracts. Two kinds of weights are introduced on the links of the built bio-term graph and are taken as important factors in the process of disambiguation. The modified Personalised PageRank (PPR) algorithm is used for performing WSD. When evaluated on the NLM-WSD and MSH-WSD test datasets, and an acronym test set, the method outperforms the widely used unsupervised ones addressing the same problem, and the average result is almost equal to that of the BlueBERT_LE-based method. In contrast, our method has no additional enhancement or training for BERT-based models. Comparative experiments validate the positive effect of links' weight on disambiguation efficiency. Last, the statistical experiments on the relation among system accuracy, the numbers of medical abstracts in the corpus, and the corresponding extracted terms suggest an excellent minimum corpus scale, when resources are limited.INDEX TERMS Biomedical informatics, biomedical natural language processing, word sense disambiguation, unified medical language system, personalised PageRank algorithm. I. INTRODUCTIONWord Sense Disambiguation (WSD) systems attempt to automatically identify the proper sense of ambiguous words in context [1], [2]. For example, WSD would aim to identify the meaning of the word ''cold'' to be ''cold temperature'' or ''common cold'' depending on the context in which it occurs. WSD is often characterized as an intermediary step in the process of understanding natural language texts [2], [3]. It is beneficial for applications in the biomedical domain, such as information extraction, automated knowledge discovery, question-answering [4], etc.The associate editor coordinating the review of this manuscript and approving it for publication was Giovanni Dimauro .The disambiguation task in the biomedical field is aimed at medical texts and terms. Combining domain knowledge can better improve the system's performance. Currently, the Unified Medical Language System (UMLS) Knowledge Sources are widely used in tasks such as disambiguation and automatic question-answering in the biomedical field. However, existing work such as [5] and [6], etc., typically involves extracting concepts (CUIs) from UMLS to build systems. In disambiguation tasks, this can lead to a mapping bottleneck when converting polysemy in biomedical documents into concepts. Therefore, we build term graphs directly, not concept graphs, the latter requires the use of term-toconcept mapping tools. This method can be considered an improvement of the existing work by Duque et al. [5], whose

show abstract

“…These example applications focus on general corpora from Wikipedia and news abstracts, respectively. In the medical domain, language models were used for multi-task information extraction from medical notes [24]. They were also used for feature extraction from Twitter posts to identify patients at risk of developing depression [25].…”

Section: Related Workmentioning

confidence: 99%

Digital-Reported Outcome from Medical Notes of Schizophrenia and Bipolar Patients Using Hierarchical BERT

Khandker,

Prince,

Chekani

et al. 2023

Information

View full text Add to dashboard Cite

Patient-reported (PRO) and clinician-reported (CRO) outcomes are assessment instruments that are completed by patients and trained healthcare professionals, respectively. A PRO is a report of the direct experience of the patient with a given disease condition. A CRO is an assessment of the condition of the patient by the healthcare provider. PROs may not be accessible to all patients, especially those suffering from severe disease conditions. CROs are time-consuming and therefore administered infrequently. In the present study, we introduce a new form of assessment, the digital-reported outcome (DRO), which is automatically derived from the medical notes of the patient. DROs have a low overhead and can be generated at each patient’s visit to complement other outcome-assessment instruments and enhance clinical decision support by identifying at-risk patients. In this study, a DRO is developed to evaluate the functional impairment in the daily activities of two cohorts of patients suffering from bipolar disorder and schizophrenia. The input of the DRO is a single medical note from the electronic medical record of the patient. This note is submitted to a hierarchical bidirectional encoder representations from transformers (BERT) model. First, a sentence-level embedding is produced for each sentence in the note using a token-level attention mechanism. Second, an embedding for the entire note is constructed using a sentence-level attention mechanism. Third, the final embedding is classified using a feed-forward neural network. The model is trained to classify patients into moderate or severe functioning impairment levels according to the general assessment of functioning (GAF) scale, a CRO instrument for the assessment of the impact of mental illness on the daily activities of the patient. The DRO is validated using medical notes that were labeled by multiple healthcare providers from different healthcare institutions. The results indicate that a general DRO is able to classify patients from the two cohorts according to the two functioning impairment levels (severe versus moderate) prior to the onset of disease with an AUC of 76%. Disease-specific DROs are only applicable after the onset of the disease and produced AUCs of nearly 85%. The methodology introduced in the present paper is practical and can support the automated monitoring of the severity of the functioning impairment of bipolar and schizophrenia patients. Extending the proposed DRO to other psychiatric conditions and types of impairments is the subject of ongoing research work.

show abstract

MT-clinical BERT: scaling clinical information extraction with multitask learning

Cited by 29 publications

References 12 publications

Recycle-BERT: Extracting Knowledge about Plastic Waste Recycling by Natural Language Processing

Recycle-BERT: Extracting Knowledge about Plastic Waste Recycling by Natural Language Processing

Weighted Co-Occurrence Bio-Term Graph for Unsupervised Word Sense Disambiguation in the Biomedical Domain

Digital-Reported Outcome from Medical Notes of Schizophrenia and Bipolar Patients Using Hierarchical BERT

Contact Info

Product

Resources

About