Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives

Akhtyamova, Liliya; Martı́nez, Paloma; Verspoor, Karin; Cardiff, John

doi:10.1109/access.2020.3018688

Cited by 21 publications

(14 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regarding the clinical domain in Spanish, we found the models Biomedical Roberta (Carrino et al, 2022) and SciELO Flair (Akhtyamova et al, 2020). In the first case, the main difference with our model is that Biomedical Roberta was trained on a corpus formed by several biomedical and clinical corpora, while we only used clinical narratives.…”

Section: Related Workmentioning

confidence: 98%

Proceedings of the 4th Clinical Natural Language Processing Workshop

2022

View full text Add to dashboard Cite

The past decade has seen tremendous progress in the field of clinical natural language processing. Driven by new algorithms and access to clinical text from electronic medical records, clinical NLP is quickly becoming a standard tool used in patient care, secondary use and medical research. At the same time, the field of NLP as a whole is undergoing a rapid transformation driven by large language models. Given these developments, it's time that we rethink the future of clinical NLP. Bio: Mark Dredze is the John C Malone Associate Professor of Computer Science at Johns Hopkins University. He is affiliated with the Malone Center for Engineering in Healthcare, the Center for Language and Speech Processing, among others. He holds a secondary appointment in the Biomedical Informatics & Data Science Section (BIDS), under the Department of Medicine (DOM), Division of General Internal Medicine (GIM) in the School of Medicine. He obtained his PhD from the University of Pennsylvania in 2009.Prof. Dredze's research develops statistical models of language with applications to social media analysis, public health and clinical informatics. Within Natural Language Processing he focuses on statistical methods for information extraction but has considered a wide range of NLP tasks, including syntax, semantics, sentiment and spoke language processing. His work in public health includes tobacco control, vaccination, infectious disease surveillance, mental health, drug use, and gun violence prevention. He also develops new methods for clinical NLP on medical records.

show abstract

Section: Related Workmentioning

confidence: 98%

Proceedings of the 4th Clinical Natural Language Processing Workshop

2022

View full text Add to dashboard Cite

show abstract

“…These embeddings, however, were not intrinsically evaluated nor compared performance-wise with other embeddings, and they were not made available for use. In another work, Akhtyamova et al ( 2020 ) used the Flair (Akbik et al, 2019 ) and BERT (Devlin et al, 2018 ) models to calculate word embeddings for the Spanish clinical domain as part of a named entity recognition (NER) task and Rojas et al ( 2022 ) computed another Flair language model from clinical narratives in Spanish. These models utilize contextualized word embeddings that take into account the word context upon embedding calculation.…”

Section: Related Workmentioning

confidence: 99%

Training and intrinsic evaluation of lightweight word embeddings for the clinical domain in Spanish

Chiu

Villena

Martin

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Resources for Natural Language Processing (NLP) are less numerous for languages different from English. In the clinical domain, where these resources are vital for obtaining new knowledge about human health and diseases, creating new resources for the Spanish language is imperative. One of the most common approaches in NLP is word embeddings, which are dense vector representations of a word, considering the word's context. This vector representation is usually the first step in various NLP tasks, such as text classification or information extraction. Therefore, in order to enrich Spanish language NLP tools, we built a Spanish clinical corpus from waiting list diagnostic suspicions, a biomedical corpus from medical journals, and term sequences sampled from the Unified Medical Language System (UMLS). These three corpora can be used to compute word embeddings models from scratch using Word2vec and fastText algorithms. Furthermore, to validate the quality of the calculated embeddings, we adapted several evaluation datasets in English, including some tests that have not been used in Spanish to the best of our knowledge. These translations were validated by two bilingual clinicians following an ad hoc validation standard for the translation. Even though contextualized word embeddings nowadays receive enormous attention, their calculation and deployment require specialized hardware and giant training corpora. Our static embeddings can be used in clinical applications with limited computational resources. The validation of the intrinsic test we present here can help groups working on static and contextualized word embeddings. We are releasing the training corpus and the embeddings within this publication1.

show abstract

“…In non-English or low-resource language. Most works in biomedical pre-trained language models are with English corpora, and a few about Chinese [292], German [25], Japanese [102,245], Spanish [6,7,145,159], Korean [109], Russian [239], Italian [28], Arabic [13,23], French [41], Portuguese [208,209] etc. For the non-English biomedical tasks, there are two mainstream solutions: a single non-English language paradigm and a multi-linguistic paradigm.…”

Section: Future Trendsmentioning

confidence: 99%

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Wang¹,

Xie²,

Pei³

et al. 2021

Preprint

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities proposes various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.CCS Concepts: • Computing methodologies → Natural language processing; Natural language generation; Neural networks; Bio-inspired approaches.

show abstract

Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives

Cited by 21 publications

References 35 publications

Proceedings of the 4th Clinical Natural Language Processing Workshop

Proceedings of the 4th Clinical Natural Language Processing Workshop

Training and intrinsic evaluation of lightweight word embeddings for the clinical domain in Spanish

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Contact Info

Product

Resources

About