Pretrained Biomedical Language Models for Clinical NLP in Spanish

Carrino, Casimiro Pio; Llop, Joan; Pàmies, Marc; Gutiérrez-Fandiño, Asier; Armengol-Estapé, Jordi; Joaquín, Silveira-Ocampo,; Valencia, Alfonso; González-Agirre, Aitor; Villegas, Marta

doi:10.18653/v1/2022.bionlp-1.19

Cited by 17 publications

(18 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nowadays, there is a strong development of contextualized word embeddings that assign dynamic representations to words based on their contexts, achieving state-of-the-art performance in multiple tasks. For the clinical domain in Spanish, relevant works include (Akhtyamova et al, 2020 ; Carrino et al, 2022 ; Rojas et al, 2022 ). These contextualized word embeddings are challenging to compute and deploy in production environments due to their demanding infrastructure needs.…”

Section: Discussionmentioning

confidence: 99%

Training and intrinsic evaluation of lightweight word embeddings for the clinical domain in Spanish

Chiu

Villena

Martin

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Resources for Natural Language Processing (NLP) are less numerous for languages different from English. In the clinical domain, where these resources are vital for obtaining new knowledge about human health and diseases, creating new resources for the Spanish language is imperative. One of the most common approaches in NLP is word embeddings, which are dense vector representations of a word, considering the word's context. This vector representation is usually the first step in various NLP tasks, such as text classification or information extraction. Therefore, in order to enrich Spanish language NLP tools, we built a Spanish clinical corpus from waiting list diagnostic suspicions, a biomedical corpus from medical journals, and term sequences sampled from the Unified Medical Language System (UMLS). These three corpora can be used to compute word embeddings models from scratch using Word2vec and fastText algorithms. Furthermore, to validate the quality of the calculated embeddings, we adapted several evaluation datasets in English, including some tests that have not been used in Spanish to the best of our knowledge. These translations were validated by two bilingual clinicians following an ad hoc validation standard for the translation. Even though contextualized word embeddings nowadays receive enormous attention, their calculation and deployment require specialized hardware and giant training corpora. Our static embeddings can be used in clinical applications with limited computational resources. The validation of the intrinsic test we present here can help groups working on static and contextualized word embeddings. We are releasing the training corpus and the embeddings within this publication1.

show abstract

Section: Discussionmentioning

confidence: 99%

Training and intrinsic evaluation of lightweight word embeddings for the clinical domain in Spanish

Chiu

Villena

Martin

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…Both of them are accessible through Hugging Face [30]. They are RoBERTa-based [31] models, trained with Spanish biomedical texts and fine-tuned using data from MEDDOPROF corpus [32]. MEDDOPROF is a public corpus consisting of 1,844 Spanish clinical case reports with annotations for occupations (i.e., occupations that provide a person with an income or livelihood), working status, and activities (i.e., non-remunerated professions); as well as annotations for to whom the occupation belongs, namely, patient, family member, health professional, or others.…”

Section: Methodsmentioning

confidence: 99%

Occupation Recognition and Exploitation in Rheumatology Clinical Notes: Employing Deep Learning Models for Named Entity Recognition and Knowledge Discovery in Electronic Health Records

Madrid-García,

Pérez-Sancristóbal,

Leticia-Leon

et al. 2024

Preprint

View full text Add to dashboard Cite

Occupation is considered a Social Determinant of Health (SDOH) and its effects have been studied at multiple levels. Although the inclusion of such data in the Electronic Health Record (EHR) is vital for the provision of clinical care, specially in rheumatology where work disability prevention is essential, occupation information is often either not routinely documented or captured in an unstructured manner within conventional EHR systems. Encouraged by recent advances in natural language processing and deep learning models, we propose the use of novel architectures (i.e., transformers) to detect occupation mentions in rheumatology clinical notes of a tertiary hospital, and to whom those occupations belongs. We also aimed to evaluate the clinical and demographic characteristics that influence the collection of this SDOH; and the association between occupation and patients' diagnosis. Bivariate and multivariate logistic regression analysis were conducted for this purpose. A Spanish pre-trained language model, RoBERTa, fine-tuned with biomedical texts was used to detect occupations. The best model achieved a F1-score of 0.725 identifying occupation mentions. Moreover, highly disabling mechanical pathology diagnoses (i.e., back pain, muscle disorders) were associated with a higher probability of occupation collection. Ultimately, we determined the professions most closely associated with thirteen categories of musculoskeletal disorders.

show abstract

“…Similarly, training models from scratch on mixed-domain data is disputed by the biomedical NLP community, especially in a low-resource setting [44]. Overall, in-domain pretraining is arguably the best option, given the availability of training data and computational resources [45][46][47].…”

Section: Pretrained Language Models For Loementioning

confidence: 99%

“…Named Entity Recognition (NER) is a foundational task for efficient information extraction. Unsurprisingly, it is known as "the most studied task in the biomedical and clinical NLP literature" [48]. Medical named entities have a more complicated structure than entities in other domains.…”

Section: Named Entity Recognition Normalization and Linking For Loementioning

confidence: 99%

Exploring the Latest Highlights in Medical Natural Language Processing across Multiple Languages: A Survey

Shaitarova,

Zaghir,

Lavelli

et al. 2023

Yearb Med Inform

View full text Add to dashboard Cite

Objectives: This survey aims to provide an overview of the current state of biomedical and clinical Natural Language Processing (NLP) research and practice in Languages other than English (LoE). We pay special attention to data resources, language models, and popular NLP downstream tasks. Methods: We explore the literature on clinical and biomedical NLP from the years 2020-2022, focusing on the challenges of multilinguality and LoE. We query online databases and manually select relevant publications. We also use recent NLP review papers to identify the possible information lacunae. Results: Our work confirms the recent trend towards the use of transformer-based language models for a variety of NLP tasks in medical domains. In addition, there has been an increase in the availability of annotated datasets for clinical NLP in LoE, particularly in European languages such as Spanish, German and French. Common NLP tasks addressed in medical NLP research in LoE include information extraction, named entity recognition, normalization, linking, and negation detection. However, there is still a need for the development of annotated datasets and models specifically tailored to the unique characteristics and challenges of medical text in some of these languages, especially low-resources ones. Lastly, this survey highlights the progress of medical NLP in LoE, and helps at identifying opportunities for future research and development in this field.

show abstract

Pretrained Biomedical Language Models for Clinical NLP in Spanish

Cited by 17 publications

References 12 publications

Training and intrinsic evaluation of lightweight word embeddings for the clinical domain in Spanish

Training and intrinsic evaluation of lightweight word embeddings for the clinical domain in Spanish

Occupation Recognition and Exploitation in Rheumatology Clinical Notes: Employing Deep Learning Models for Named Entity Recognition and Knowledge Discovery in Electronic Health Records

Exploring the Latest Highlights in Medical Natural Language Processing across Multiple Languages: A Survey

Contact Info

Product

Resources

About