Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives

Abstract

Background: In the Big Data era there is an increasing need to fully exploit and analyse the huge quantity of information available about health. Natural Language Processing (NLP) technologies can contribute to extract relevant information from unstructured data contained in Electronic Health Records (EHR) such as clinical notes, patient’s discharge summaries and radiology reports among others. Extracted information could help in health-related decision making processes. Named entity recognition (NER) devoted to detect important concepts in texts (diseases, symptoms, drugs, etc.) is a crucial task in information extraction processes especially in languages other than English. In this work, we develop a deep learning-based NLP pipeline for biomedical entity extraction in Spanish clinical narrative. Methods: We explore the use of contextualized word embeddings to enhance named entity recognition in Spanish language clinical text, particularly of pharmacological substances, compounds, and proteins. Various combinations of word and sense embeddings were tested on the evaluation corpus of the PharmacoNER 2019 task, the Spanish Clinical Case Corpus (SPACCC). This data set consists of clinical case sections derived from open access Spanish-language medical publications. Results: NER system integrates in-domain pre-trained Flair and FastText word embeddings, byte-pairwise encoded and the bi-LSTM-based character word embeddings. The system yielded the best performance measure with F-score of 90.84%. Error analysis showed that the main source of errors for the best model is the newly detected false positive entities with the half of that amount of errors belonged to longer than the actual ones detected entities. Conclusions: Our study shows that our deep-learning-based system with domain-specific contextualized embeddings coupled with stacking of complementary embeddings yields superior performance over the system with integrated standard and general-domain word embeddings. With this system, we achieve performance competitive with the state-of-the-art.

Training and intrinsic evaluation of lightweight word embeddings for the clinical domain in Spanish

“…Nowadays, there is a strong development of contextualized word embeddings that assign dynamic representations to words based on their contexts, achieving state-of-the-art performance in multiple tasks. For the clinical domain in Spanish, relevant works include (Akhtyamova et al, 2020;Carrino et al, 2022;Rojas et al, 2022). These contextualized word embeddings are challenging to compute and deploy in production environments due to their demanding infrastructure needs.…”

“…In sum, we have worked with a corpus of 235 million tokens, which is a good number compared to, for example, the 86 million tokens collected by Akhtyamova et al (2020). Nevertheless, our corpus is still way below the 13.5 billion tokes used to calculate BERT for the clinical domain in English (Lee et al, 2020).…”

“…These embeddings, however, were not intrinsically evaluated nor compared performance-wise with other embeddings, and they were not made available for use. In another work, Akhtyamova et al (2020) used the Flair (Akbik et al, 2019) and BERT (Devlin et al, 2018) models to calculate word embeddings for the Spanish clinical domain as part of a named entity recognition (NER) task and Rojas et al (2022) computed another Flair language model from clinical narratives in Spanish. These models utilize contextualized word embeddings that take into account the word context upon embedding calculation.…”

Arabic Questions Classification Using Modified TF-IDF

“…It allows capturing the similarity between individual word vectors, thus providing information on the underlying word meanings [38]. Several pretrained word embedding models have been developed, such as word2vec, global vectors for word representation (GloVe) and FastText [39].…”