ELECTRAMed: a new pre-trained language representation model for biomedical NLP

Miolo, Giacomo; Mantoan, Giulio; Orsenigo, Carlotta

doi:10.48550/arxiv.2104.09585

Cited by 3 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Consequently, they propose PubMedBERT which is pre-trained on 14M PubMed abstracts from scratch. Similarly, [20] pre-train on 28M data as in [8] also from scratch, using the more advanced ELECTRA model. All these works have shown improved performance on plenty of biomedical literature language processing tasks compared to the original BERT pretrained on general domain, while none of them is for biomedical generation tasks.…”

Section: Pre-trained Language Models In Biomedical Domainmentioning

confidence: 99%

BioGPT: generative pre-trained transformer for biomedical text generation and mining

Luo

Sun

Xia

et al. 2022

Briefings in Bioinformatics

409

159

View full text Add to dashboard Cite

Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.

show abstract

Section: Pre-trained Language Models In Biomedical Domainmentioning

confidence: 99%

BioGPT: generative pre-trained transformer for biomedical text generation and mining

Luo

Sun

Xia

et al. 2022

Briefings in Bioinformatics

409

159

View full text Add to dashboard Cite

show abstract

“…The authors of the papers [5][6][7] involve architectures based on sequential and transformer networks to solve analytical medical problems, which can be applied to our task for the transfer of knowledge in the cross-domain field.…”

Section: Literature Reviewmentioning

confidence: 99%

Using Deep Learning to Generate and Classify Thyroid Cytopathology Reports According to The Bethesda System

Diuldin,

Makanov,

Shifman

et al. 2024

RIA

View full text Add to dashboard Cite

The purpose of this paper is to study approaches to the intellectual processing of Russianlanguage textual medical information concerning the thyroid cytopathology description to solve the issues of their classification and generation of the text of the medical report, as well as augmentation of descriptions in their acute shortage. Over the past decade, the field of biomedicine has not undergone significant changes. Approaches to analyzing patients' problems are mostly based on manual processing and expert knowledge of doctors. The paper considers the creation of a machine-learning pipeline containing a full cycle of data preprocessing and model training in the field of thyroid nodules fine-needle aspiration classification according to the Bethesda thyroid cytopathology reporting system. Sequential and transformer neural networks were used to design the architecture of deep learning models. The paper proposes approaches for cleaning and preprocessing raw medical descriptions to the required type. The obtained results show that sequential neural networks have greater accuracy on small data sets, and transformation architectures are superior to others when generating cytopathological reports on large amounts of data. The solution obtained in the study can be used as an additional reference tool for thyroid cytologists.

show abstract

“…(3) Pre-training language models are an open research area that guide the acquisition of striking gains on several natural language processing tasks. Some of the language representation models such as BERT (Devlin, Chang, Lee, & Toutanova, 2018) and ELMO have been trained generally for general domain datasets, and they perform weakly on domain-specific text mining tasks (Lee et al, 2020), thereby leading to the development of domain-specific Khadivi and Sato: A Bibliometric Study of NLP language models like Bio-BERT and ELECTRAMed (Miolo, Mantoan, & Orsenigo, 2021), which are applicable for biomedical text mining, or BERT_SE (Fávero & Casanova, 2021), which is used for noticing software engineering vocabulary, which was initiated a few years ago. Additionally, in recent years, BERT has been used for fake news detection.…”

Section: Discovering Research Trends and Future Directionmentioning

confidence: 99%

A Bibliometric Study of Natural Language Processing Using Dimensions Database: Development, Research Trend, and Future Research Directions

Khadivi,

Sato

2023

J Data Sci. Info. Citation Studies

View full text Add to dashboard Cite

A division of artificial intelligence known as natural language processing is thought to be a rapidly developing subject. NLP methods enable computers to comprehend human words and translate unstructured data for use by systems. The Dimensions database, which contains papers on natural language processing from 2002 to 2021, is thoroughly examined bibliometrically in this paper. It is the first attempt to perform bibliometric analysis in the NLP area in Dimensions database papers. The information is examined using bibliometric growth indicators, such as growth rate, relative growth rate, doubling time, degree of collaboration, and collaboration index. In addition, the VOSviewer software is used to create bibliometric networks. The primary purposes of this research study include detecting the NLP development pattern, discovering the distribution of co-authorship, influential authors, productive geographical distribution, keyword analysis, highly cited publications, finding recent research hotspots, and determining future research directions based on the challenges that exist in each of the research trends.

show abstract

ELECTRAMed: a new pre-trained language representation model for biomedical NLP

Cited by 3 publications

References 17 publications

BioGPT: generative pre-trained transformer for biomedical text generation and mining

BioGPT: generative pre-trained transformer for biomedical text generation and mining

Using Deep Learning to Generate and Classify Thyroid Cytopathology Reports According to The Bethesda System

A Bibliometric Study of Natural Language Processing Using Dimensions Database: Development, Research Trend, and Future Research Directions

Contact Info

Product

Resources

About