BioMegatron: Larger Biomedical Domain Language Model

Shin, Hoo-Chang; Zhang, Yang; Bakhturina, Evelina; Puri, Raul; Patwary, Mostofa; Shoeybi, Mohammad; Mani, Raghav

doi:10.18653/v1/2020.emnlp-main.379

Cited by 81 publications

(60 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Domain-specific pretraining on biomedical corpora (e.g. BIOBERT, Lee et al 2020 and BIOMEGA-TRON, Shin et al 2020) have made much progress in biomedical text mining tasks. Nonetheless, representing medical entities with the existing SOTA pretrained MLMs (e.g.…”

Section: Pubmedbert + Sapbertmentioning

confidence: 99%

Self-Alignment Pretraining for Biomedical Entity Representations

Liu¹,

Shareghi²,

Meng³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

160

108

View full text Add to dashboard Cite

Despite the widespread success of selfsupervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we propose SAPBERT, a pretraining scheme that selfaligns the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with 4M+ concepts. In contrast with previous pipelinebased hybrid systems, SAPBERT offers an elegant one-model-for-all solution to the problem of medical entity linking (MEL), achieving a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific domain, we achieve SOTA even without taskspecific supervision. With substantial improvement over various domain-specific pretrained MLMs such as BIOBERT, SCIBERT and PUB-MEDBERT, our pretraining scheme proves to be both effective and robust. 1

show abstract

Section: Pubmedbert + Sapbertmentioning

confidence: 99%

Self-Alignment Pretraining for Biomedical Entity Representations

Liu¹,

Shareghi²,

Meng³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

160

108

View full text Add to dashboard Cite

show abstract

“…BioMegaTron 345m (Shin et al, 2020) is a largescale model (345m parameters) by NVIDIA based on MegaTron architecture. .…”

Section: Biomedical Language Modelsmentioning

confidence: 99%

“…Our choices of downstream biomedical tasks are similar to (Shin et al, 2020). For Named Entity Recognition (NER) and Relation Extraction (RE), we generate our training, development, and test data using the same script that PubMedBERT uses (Gu et al, 2021).…”

Section: Downstream Tasksmentioning

confidence: 99%

“…Furthermore, our evaluation shows that ELECTRA B++ model achieve state-of-the-art result on BioASQ for base-scale models. We attribute this performance to the fact that we use (Gu et al, 2021), BioMegaTron (Shin et al, 2020), BioRoBERTa L (Lewis et al, 2020). We generate QA results for all models, except that we use reported results for BioMegaTron, BioBERT (Shin et al, 2020), RoBERTa B .…”

Section: Question Answeringmentioning

confidence: 99%

“…We attribute this performance to the fact that we use (Gu et al, 2021), BioMegaTron (Shin et al, 2020), BioRoBERTa L (Lewis et al, 2020). We generate QA results for all models, except that we use reported results for BioMegaTron, BioBERT (Shin et al, 2020), RoBERTa B . BioMegaTron uses sub-tokens evaluation for NER tasks rather than whole-entity evaluation and uses different pre-processed data set for ChemProt task.…”

Section: Question Answeringmentioning

confidence: 99%

See 2 more Smart Citations

BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA

Alrowili¹,

Shanker²

2021

Proceedings of the 20th Workshop on Biomedical Language Processing

View full text Add to dashboard Cite

The impact of design choices on the performance of biomedical language models recently has been a subject for investigation. In this paper, we empirically study biomedical domain adaptation with large transformer models using different design choices. We evaluate the performance of our pretrained models against other existing biomedical language models in the literature. Our results show that we achieve state-of-the-art results on several biomedical domain tasks despite using similar or less computational cost compared to other models in the literature. Our findings highlight the significant effect of design choices on improving the performance of biomedical language models.

show abstract

Artificial intelligence for neurodegenerative experimental models

Marzi,

Schilder,

Nott

et al. 2023

Alzheimer's & Dementia

View full text Add to dashboard Cite

INTRODUCTIONExperimental models are essential tools in neurodegenerative disease research. However, the translation of insights and drugs discovered in model systems has proven immensely challenging, marred by high failure rates in human clinical trials.METHODSHere we review the application of artificial intelligence (AI) and machine learning (ML) in experimental medicine for dementia research.RESULTSConsidering the specific challenges of reproducibility and translation between other species or model systems and human biology in preclinical dementia research, we highlight best practices and resources that can be leveraged to quantify and evaluate translatability. We then evaluate how AI and ML approaches could be applied to enhance both cross‐model reproducibility and translation to human biology, while sustaining biological interpretability.DISCUSSIONAI and ML approaches in experimental medicine remain in their infancy. However, they have great potential to strengthen preclinical research and translation if based upon adequate, robust, and reproducible experimental data.Highlights There are increasing applications of AI in experimental medicine. We identified issues in reproducibility, cross‐species translation, and data curation in the field. Our review highlights data resources and AI approaches as solutions. Multi‐omics analysis with AI offers exciting future possibilities in drug discovery.

show abstract

BioMegatron: Larger Biomedical Domain Language Model

Cited by 81 publications

References 16 publications

Self-Alignment Pretraining for Biomedical Entity Representations

Self-Alignment Pretraining for Biomedical Entity Representations

BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA

Artificial intelligence for neurodegenerative experimental models

Contact Info

Product

Resources

About