BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition

Naseem, Usman; Khushi, Matloob; Reddy, Vinay P; Rajendran, Senthilkumar; Razzak, Imran; Kim, Jin Man

doi:10.48550/arxiv.2009.09223

Cited by 5 publications

(8 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BioBERT [121], BlueBERT [179], SciBERT [16], BioELMo [93], PubMedBERT [66], BioMegatron [215],Yuan et al [286], Alsentze et al [10], Singh et al [206], Zhu et al [301], Si et al [216], Sheikhshab et al [213], Khan et al [103], Giorgi et al [63], Naseem [168], Gao et al [59], Poerner et al [186], Sun et al [229], for Spanish [6,75,159],…”

Section: Named Entity Recognitionmentioning

confidence: 99%

“…It utilized the BioBert as the transformer encoder layer and the multiple data sets in the task-specific layers. With the development of general domain language models, Naseem [168] proposed an effective domain-specific language model bioALBERT trained on the biomedical domain corpora (PubMed abstracts and PMC full-text articles) for biomedical named entity recognition. Giorgi et al [63] proposed the end-to-end model for jointly extracting named entities and their relations using the pre-trained language model BERT.…”

Section: Named Entity Recognition Biomedical Named Entity Recognition...mentioning

confidence: 99%

See 1 more Smart Citation

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Wang¹,

Xie²,

Pei³

et al. 2021

Preprint

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities proposes various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.CCS Concepts: • Computing methodologies → Natural language processing; Natural language generation; Neural networks; Bio-inspired approaches.

show abstract

Section: Named Entity Recognitionmentioning

confidence: 99%

Section: Named Entity Recognition Biomedical Named Entity Recognition...mentioning

confidence: 99%

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Wang¹,

Xie²,

Pei³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Table 1 summarises a number of datasets previously used to evaluate Pre-trained LMs on various BioNLP tasks. Our previous preliminary work has shown the potential of designing a customised domain-specific LM outperforming SOTA in NER tasks 16 .…”

Section: Background and Summarymentioning

confidence: 99%

Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT

Naseem¹,

Dunn²,

Khushi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The availability of biomedical text data and advances in natural language processing (NLP) have made new applications in biomedical NLP possible. Language models trained or fine-tuned using domain-specific corpora can outperform general models, but work to date in biomedical NLP has been limited in terms of corpora and tasks. We present BioALBERT, a domain-specific adaptation of A Lite Bidirectional Encoder Representations from Transformers (ALBERT), trained on biomedical (PubMed and PubMed Central) and clinical (MIMIC-III) corpora and fine-tuned for 6 different tasks across 20 benchmark datasets. Experiments show that BioALBERT outperforms the state-of-the-art on named-entity recognition (+11.09% BLURB score improvement), relation extraction (+0.80% BLURB score), sentence similarity (+1.05% BLURB score), document classification (+0.62% F1-score), and question answering (+2.83% BLURB score). It represents a new state-of-the-art in 17 out of 20 benchmark datasets. By making BioALBERT models and data available, our aim is to help the biomedical NLP community avoid computational costs of training and establish a new set of baselines for future efforts across a broad range of biomedical NLP tasks.

show abstract

“…One limitation of transformer-based models is the inability to capture information that is specific to a domain. To improve results in specialized domains, several transformer-based LMs such as Biomedical BERT (BioBERT) [18], Biomedical A Lite Bidirectional Encoder Representations from Transformers (BioALBERT) [32], Twitter BERT (BertTweet) [38] and Covid-BERT (CT-BERT) [30], were trained on domainspecific corpora using the same unsupervised training method used in general models. Given how domain-specific LMs have improved performance for specific NLP tasks, we expect that domain-specific LMs should improve the vaccine sentiment classification task.…”

Section: Introductionmentioning

confidence: 99%

Classifying vaccine sentiment tweets by modelling domain-specific representation and commonsense knowledge into context-aware attentive GRU

Naseem¹,

Khushi²,

Kim³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Vaccines are an important public health measure, but vaccine hesitancy and refusal can create clusters of low vaccine coverage and reduce the effectiveness of vaccination programs. Social media provides an opportunity to estimate emerging risks to vaccine acceptance by including geographical location and detailing vaccine-related concerns. Methods for classifying social media posts, such as vaccine-related tweets, use language models (LMs) trained on general domain text. However, challenges to measuring vaccine sentiment at scale arise from the absence of tonal stress and gestural cues and may not always have additional information about the user, e.g., past tweets or social connections. Another challenge in LMs is the lack of 'commonsense' knowledge that are apparent in users' metadata, i.e., emoticons, positive and negative words etc. In this study, to classify vaccine sentiment tweets with limited information, we present a novel end-to-end framework consisting of interconnected components that use domain-specific LM trained on vaccine-related tweets and models commonsense knowledge into a bidirectional gated recurrent network (CK-BiGRU) with context-aware attention. We further leverage syntactical, user metadata and sentiment information to capture the sentiment of a tweet. We experimented using two popular vaccine-related Twitter datasets and demonstrate that our proposed approach outperforms state-of-the-art models in identifying pro-vaccine, anti-vaccine and neutral tweets.

show abstract

BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition

Cited by 5 publications

References 22 publications

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT

Classifying vaccine sentiment tweets by modelling domain-specific representation and commonsense knowledge into context-aware attentive GRU

Contact Info

Product

Resources

About