BioVAE: a pre-trained latent variable language model for biomedical text mining

Abstract

Summary Large scale pre-trained language models (PLMs) have advanced state-of-the-art (SOTA) performance on various biomedical text mining tasks. The power of such PLMs can be combined with the advantages of deep generative models. These are examples of these combinations. However, they are trained only on general domain text, and biomedical models are still missing. In this work, we describe BioVAE, the first large scale pre-trained latent variable language model for the biomedical domain, which uses the OPTIMUS framework to train on large volumes of biomedical text. The model shows SOTA performance on several biomedical text mining tasks when compared to existing publicly available biomedical PLMs. Additionally, our model can generate more accurate biomedical sentences than the original OPTIMUS output. Availability Our source code and pre-trained models are freely available: https://github.com/aistairc/BioVAE Supplementary information Supplementary data are available at Bioinformatics online.

RDscan: Extracting RNA-disease relationship from literature Using pre-training model

“…Especially with the development of large language models like BERT and GPT, the extraction of specific information has become more efficient [29]. Currently, pre-trained models tailored for biomedical literature, such as bioBERT [30], BioVAE [31], bioGPT [32] and Bioformer [33] have been successively developed.…”