BioGPT: generative pre-trained transformer for biomedical text generation and mining

Abstract

Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.

Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine

“… 7 BioGPT, a LLM developed by Microsoft and trained on biomedical data, has achieved human parity in certain tasks of biomedical text generation and mining. 8 Similar outcomes have been reported with other state-of-the-art LLMs for standardized tests in law and education. This capability, paired with the conversational ease of human-machine interaction and the sheer limitless scope of prompt complexity and variability which LLMs can handle is an impressive accomplishment.…”

Assessing the Value of ChatGPT for Clinical Decision Support Optimization

“…[34] Another example is BioGPT, a GPT-like model trained on PubMed articles. [35] In addition, UpToDate is an important source of CDS content for alert development and should also be considered for adding to the language model. Second, based on the Reinforcement Learning from Human Feedback (RLHF) framework, researchers could train a model specifically for this task on the improved language model.…”

Discovering Themes in Deep Brain Stimulation Research Using Explainable Artificial Intelligence

“…The code is available, though the reader should know that topic modeling is not the most sophisticated language model available. Future iterations of language models about deep brain stimulation could use a more advanced approach, such as a generative pre-trained transformer model validated by experts in the field [ 50 ].…”