Results of the Seventh Edition of the BioASQ Challenge

Nentidis, Anastasios; Bougiatiotis, Konstantinos; Krithara, Anastasia; Παλιούρας, Γεώργιος

doi:10.1007/978-3-030-43887-6_51

Cited by 35 publications

(22 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Aside from the vocabulary issue mentioned earlier, neural network training uses non-convex optimization, which means that continual pretraining may not be able to completely undo suboptimal initialization from the general-domain language model. BioBERT [34] SciBERT [8] BLUE [45] BLURB BC5-chem [35] BC5-disease [35] NCBI-disease [18] -BC2GM [53] --JNLPBA [27] --EBM PICO [44] --ChemProt [31] DDI [21] -GAD [11] --BIOSSES [54] --HoC [20] --PubMedQA [25] ---BioASQ [42] --…”

Section: Domain-specificmentioning

confidence: 99%

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tinn

Cheng

et al. 2021

ACM Trans. Comput. Healthcare

871

514

View full text Add to dashboard Cite

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .

show abstract

Section: Domain-specificmentioning

confidence: 99%

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tinn

Cheng

et al. 2021

ACM Trans. Comput. Healthcare

871

514

View full text Add to dashboard Cite

show abstract

“…Additionally we evaluate on CoNLL-03 (Tjong Kim Sang and De Meulder, 2003) named entity recognition (NER), and SQuAD 1.1 (Rajpurkar et al, 2016) question answering (QA). To demon-strate domain shift we evaluate using BC5CDR , Chemprot (Krallinger et al, 2017) and BioASQ (Nentidis et al, 2019) which are biomedical NER, relation extraction (RE), and QA tasks respectively. The first dataset is from the 2015 CDR challenge for identifying chemicals and diseases expertly annotated from Pubmed abstracts 4 .…”

Section: Evaluation Datamentioning

confidence: 99%

An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training

Arumae

Sun

Bhatia

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Pre-training large language models has become a standard in the natural language processing community. Such models are pretrained on generic data (e.g. BookCorpus and English Wikipedia) and often fine-tuned on tasks in the same domain. However, in order to achieve state-of-the-art performance on out of domain tasks such as clinical named entity recognition and relation extraction, additional in domain pre-training is required. In practice, staged multi-domain pre-training presents performance deterioration in the form of catastrophic forgetting (CF) when evaluated on a generic benchmark such as GLUE. In this paper we conduct an empirical investigation into known methods to mitigate CF. We find that elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. Furthermore, we explore gradient and latent clustering based data selection techniques to improve coverage when using elastic weight consolidation and experience replay methods.

show abstract

“…ChemProt ( Kringelum et al, 2016 ); and a QA task, i.e. BioASQ ( Nentidis et al , 2019 ). We follow the same evaluation settings used in Lee et al (2020) and Beltagy et al (2019) .…”

Section: Resultsmentioning

confidence: 99%

“…Large-scale pre-trained language models (PLMs) ( Beltagy et al , 2019 ; Lee et al , 2020 ) have shown state-of-the-art (SOTA) performance on various biomedical text mining tasks. These models provide contextualized representations, learned from large volumes of biomedical text, which then can be easily applied to achieve SOTA on downstream tasks such as named entity recognition (NER), relation extraction (REL) and question answering (QA) ( Kim et al , 2019 ; Lin et al , 2019 ; Nentidis et al , 2019 ).…”

Section: Introductionmentioning

confidence: 99%

BioVAE: a pre-trained latent variable language model for biomedical text mining

2021

View full text Add to dashboard Cite

Summary Large scale pre-trained language models (PLMs) have advanced state-of-the-art (SOTA) performance on various biomedical text mining tasks. The power of such PLMs can be combined with the advantages of deep generative models. These are examples of these combinations. However, they are trained only on general domain text, and biomedical models are still missing. In this work, we describe BioVAE, the first large scale pre-trained latent variable language model for the biomedical domain, which uses the OPTIMUS framework to train on large volumes of biomedical text. The model shows SOTA performance on several biomedical text mining tasks when compared to existing publicly available biomedical PLMs. Additionally, our model can generate more accurate biomedical sentences than the original OPTIMUS output. Availability Our source code and pre-trained models are freely available: https://github.com/aistairc/BioVAE Supplementary information Supplementary data are available at Bioinformatics online.

show abstract

Results of the Seventh Edition of the BioASQ Challenge

Cited by 35 publications

References 42 publications

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training

BioVAE: a pre-trained latent variable language model for biomedical text mining

Contact Info

Product

Resources

About