Active Learning with Deep Pre-trained Models for Sequence Tagging of Clinical and Biomedical Texts

Shelmanov, Artem; Härmä, Aki; Kireev, Danil; Khromov, Nikita; Panchenko, Alexander; Fedulova, Irina; Dylov, Dmitry V.

doi:10.1109/bibm47256.2019.8983157

Cited by 23 publications

(16 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep pre-trained models are evaluated in the AL setting for NER by Shelmanov et al (2019). However, they perform the evaluation only on the specific biomedical datasets and do not consider the Bayesian query strategies.…”

Section: Related Workmentioning

confidence: 99%

Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

Shelmanov

Puzyrev²,

Kupriyanova³

et al. 2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

Self Cite

View full text Add to dashboard Cite

Annotating training data for sequence tagging of texts is usually very time-consuming. Recent advances in transfer learning for natural language processing in conjunction with active learning open the possibility to significantly reduce the necessary annotation budget. We are the first to thoroughly investigate this powerful combination for the sequence tagging task. We conduct an extensive empirical study of various Bayesian uncertainty estimation methods and Monte Carlo dropout options for deep pretrained models in the active learning framework and find the best combinations for different types of models. Besides, we also demonstrate that to acquire instances during active learning, a full-size Transformer can be substituted with a distilled version, which yields better computational performance and reduces obstacles for applying deep active learning in practice.

show abstract

Section: Related Workmentioning

confidence: 99%

Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

Shelmanov

Puzyrev²,

Kupriyanova³

et al. 2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, this work focuses on a single task, and does not address the effect of small and imbalanced data. Additionally, Shelmanov et al (2019) and Liu et al (2020) focused on particular variants of BERT (BioBERT and BERT-CRF) and studied a single or two specific tasks, with a small collection of AL strategies. To the best of our knowledge, this work is the first to systematically explore advanced strategies like Core-Set (Sener and Savarese, 2017), Dropout (Gal and Ghahramani, 2016), Expected Gradient Length (Huang et al, 2016) and Discriminative Active Learning (Gissin and Shalev-Shwartz, 2019) for BERT, in various settings and a diversity of tasks.…”

Section: Related Workmentioning

confidence: 99%

Active Learning for BERT: An Empirical Study

Ein-Dor

Halfon

Gera

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Real world scenarios present a challenge for text classification, since labels are usually expensive and the data is often characterized by class imbalance. Active Learning (AL) is a ubiquitous paradigm to cope with data scarcity. Recently, pre-trained NLP models, and BERT in particular, are receiving massive attention due to their outstanding performance in various NLP tasks. However, the use of AL with deep pre-trained models has so far received little consideration. Here, we present a large-scale empirical study on active learning techniques for BERT-based classification, addressing a diverse set of AL strategies and datasets. We focus on practical scenarios of binary text classification, where the annotation budget is very small, and the data is often skewed. Our results demonstrate that AL can boost BERT performance, especially in the most realistic scenario in which the initial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class. We release our research framework, aiming to facilitate future research along the lines explored here.

show abstract

“…Finally, more “elaborate” methods have been used to extend the fine-tuning process and push benchmark performances further. Examples include the use of active learning on top of pre-trained BERT models [ 45 ], complementing the base model with a transfer learning framework [ 46 ], or a graph NN architecture [ 47 ].…”

Section: Resultsmentioning

confidence: 99%

A Review of Recent Work in Transfer Learning and Domain Adaptation for Natural Language Processing of Electronic Health Records

et al. 2021

View full text Add to dashboard Cite

Summary Objectives: We survey recent work in biomedical NLP on building more adaptable or generalizable models, with a focus on work dealing with electronic health record (EHR) texts, to better understand recent trends in this area and identify opportunities for future research. Methods: We searched PubMed, the Institute of Electrical and Electronics Engineers (IEEE), the Association for Computational Linguistics (ACL) anthology, the Association for the Advancement of Artificial Intelligence (AAAI) proceedings, and Google Scholar for the years 2018-2020. We reviewed abstracts to identify the most relevant and impactful work, and manually extracted data points from each of these papers to characterize the types of methods and tasks that were studied, in which clinical domains, and current state-of-the-art results. Results: The ubiquity of pre-trained transformers in clinical NLP research has contributed to an increase in domain adaptation and generalization-focused work that uses these models as the key component. Most recently, work has started to train biomedical transformers and to extend the fine-tuning process with additional domain adaptation techniques. We also highlight recent research in cross-lingual adaptation, as a special case of adaptation. Conclusions: While pre-trained transformer models have led to some large performance improvements, general domain pre-training does not always transfer adequately to the clinical domain due to its highly specialized language. There is also much work to be done in showing that the gains obtained by pre-trained transformers are beneficial in real world use cases. The amount of work in domain adaptation and transfer learning is limited by dataset availability and creating datasets for new domains is challenging. The growing body of research in languages other than English is encouraging, and more collaboration between researchers across the language divide would likely accelerate progress in non-English clinical NLP.

show abstract

Active Learning with Deep Pre-trained Models for Sequence Tagging of Clinical and Biomedical Texts

Cited by 23 publications

References 14 publications

Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

Active Learning for BERT: An Empirical Study

A Review of Recent Work in Transfer Learning and Domain Adaptation for Natural Language Processing of Electronic Health Records

Contact Info

Product

Resources

About