Clinical concept extraction using transformers

Yang, Xi; Bian, Jiang; Hogan, William R.; Wu, Yonghui

doi:10.1093/jamia/ocaa189

Cited by 110 publications

(66 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For NLU tasks, networks in the literature on biomedical texts span from convolutional neural networks [ 41 ] to transformers (e.g. BERT and its variants) [ 51 ].…”

Section: Framework and Discussionmentioning

confidence: 99%

A Proposed Framework for Machine Learning-Aided Triage in Public Specialty Ophthalmology Clinics in Hong Kong

Vardhanabhuti

Tsougenis

et al. 2021

Ophthalmol Ther

View full text Add to dashboard Cite

“…For NLU tasks, networks in the literature on biomedical texts span from convolutional neural networks [ 41 ] to transformers (e.g. BERT and its variants) [ 51 ].…”

Section: Framework and Discussionmentioning

confidence: 99%

A Proposed Framework for Machine Learning-Aided Triage in Public Specialty Ophthalmology Clinics in Hong Kong

Vardhanabhuti

Tsougenis

et al. 2021

Ophthalmol Ther

View full text Add to dashboard Cite

“…Thus, the results obtained in this work for automatic clinical coding in Spanish support the hypothesis asserting that, when adapted to the specificities of the clinical domain, transformer-based models outperformed their original nonspecific domain versions on downstream medical tasks. Although this hypothesis was already explored in previous works [41]- [43], its validity has only been demonstrated for clinical NLP tasks in the English language, for which a considerable amount of standardized and curated medical linguistic resources is publicly available. In contrast, in this study, we have experimentally shown the effectiveness of the clinical domain adaptation of transformers when applied to small-data NLP tasks in a non-English language with limited textual resources.…”

Section: A Domain-specific Modelsmentioning

confidence: 99%

“…We mainly focused on clinical coding, which is a document-level NLP task. However, the domain-specific transformers examined in this work could also be applied to word-level NLP problems, such as NER tasks, for which transformer-based models have been shown to achieve SOTA performance in the clinical NLP domain [41]- [43], mainly for the English language. Additionally, future studies should pay special attention to model interpretability, as though a few efforts have already been made in this particular area, most of DL models are still regarded as "black-boxes".…”

Section: B Fine-tuning Approachmentioning

confidence: 99%

Transformers for Clinical Coding in Spanish

et al. 2021

View full text Add to dashboard Cite

Automatic clinical coding is an essential task in the process of extracting relevant information from unstructured documents contained in electronic health records (EHRs). However, most research in the development of computer-based methods for clinical coding focuses on texts written in English due to the limited availability of medical linguistic resources in languages other than English. With nearly 500 million native speakers, there is a worldwide interest in processing healthcare texts in Spanish. In this study, we systematically analyzed transformer-based models for automatic clinical coding in Spanish. Using a transfer-learning-based approach, the three existing transformer architectures that support the Spanish language, namely, multilingual BERT (mBERT), BETO and XLM-RoBERTa (XLM-R), were first pretrained on a corpus of real-world oncology clinical cases with the goal of adapting transformers to the particularities of Spanish medical texts. The resulting models were fine-tuned on three distinct clinical coding tasks, following a multilabel sentence classification strategy. For each analyzed transformer, the domain-specific version outperformed the original general domain model across those tasks. Moreover, the combination of the developed strategy with an ensemble approach leveraging the predictive capacities of the three distinct transformers yielded the best obtained results, with MAP scores of 0.662, 0.544 and 0.884 on CodiEsp-D, CodiEsp-P and Cantemist-Coding shared tasks, which remarkably improved the previous state-of-the-art performance by 11.6%, 10.3% and 4.4%, respectively. We publicly release the mBERT, BETO and XLMR transformers adapted to the Spanish clinical domain at https://github.com/guilopgar/ClinicalCodingTransformerES, providing the clinical natural language processing community with advanced deep learning methods for performing medical coding and other tasks in the Spanish clinical domain.INDEX TERMS Clinical coding, deep learning, natural language processing, text classification, transformers.

show abstract

“…Among these new context-aware language models, the Transformer [24] has undoubtedly stood out as the new deep learning SOTA architecture in the field of NLP. BERT [6], RoBERTa [13] and XLM-R [5] are examples of transformer-based models that have become the new SOTA for question answering, text summarization or NER tasks, also in the field of clinical NLP [21,1,28]. One of the main characteristics of the Transformer architecture is the self-attention mechanism it uses, which allows the model to parallelize a large part of the network architecture, increasing computing efficiency.…”

Section: Transformer-based Modelsmentioning

confidence: 99%

Detection of Tumor Morphology Mentions in Clinical Reports in Spanish Using Transformers

López-García

Jerez

Ribelles

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The aim of this study is to systematically examine the performance of transformer-based models for the detection of tumor morphology mentions in clinical documents in Spanish. For this purpose, we analyzed 3 transformer models supporting the Spanish language, namely multilingual BERT, BETO and XLM-RoBERTa. By means of a transferlearning-based approach, the models were first pretrained on a collection of real-world oncology clinical cases with the goal of adapting transformers to the distinctive features of the Spanish oncology domain. The resulting models were further fine-tuned on the Cantemist-NER task, addressing the detection of tumor morphology mentions as a multi-class sequence-labeling problem. To evaluate the effectiveness of the proposed approach, we compared the obtained results by the domain-specific version of the examined transformers with the performance achieved by the general-domain version of the models. The results obtained in this paper empirically demonstrated that, for every analyzed transformer, the clinical version outperformed the corresponding general-domain model on the detection of tumor morphology mentions in clinical case reports in Spanish. Additionally, the combination of the transfer-learning-based approach with an ensemble strategy exploiting the predictive capabilities of the distinct transformer architectures yielded the best obtained results, achieving a precision value of 0.893, a recall of 0.887 and an F1-score of 0.89, which remarkably surpassed the prior state-of-the-art performance for the Cantemist-NER task.

show abstract

Clinical concept extraction using transformers

Cited by 110 publications

References 48 publications

A Proposed Framework for Machine Learning-Aided Triage in Public Specialty Ophthalmology Clinics in Hong Kong

A Proposed Framework for Machine Learning-Aided Triage in Public Specialty Ophthalmology Clinics in Hong Kong

Transformers for Clinical Coding in Spanish

Detection of Tumor Morphology Mentions in Clinical Reports in Spanish Using Transformers

Contact Info

Product

Resources

About