Probing Biomedical Embeddings from Language Models

Abstract

Contextualized word embeddings derived from pre-trained language models (LMs) show significant improvements on downstream NLP tasks. Pre-training on domain-specific corpora, such as biomedical articles, further improves their performance. In this paper, we conduct probing experiments to determine what additional information is carried intrinsically by the in-domain trained contextualized embeddings. For this we use the pre-trained LMs as fixed feature extractors and restrict the downstream task models to not have additional sequence modeling layers. We compare BERT (Devlin et al., 2018), ELMo (Peters et al., 2018a), BioBERT (Lee et al., 2019) and BioELMo, a biomedical version of ELMo trained on 10M PubMed abstracts. Surprisingly, while fine-tuned BioBERT is better than BioELMo in biomedical NER and NLI tasks, as a fixed feature extractor BioELMo outperforms BioBERT in our probing tasks. We use visualization and nearest neighbor analysis to show that better encoding of entity-type and relational information leads to this superiority.

PubMedQA: A Dataset for Biomedical Research Question Answering

“…ESIM with BioELMo: Following the state-ofthe-art recurrent architecture of NLI (Peters et al, 2018), we use pre-trained biomedical contextualized embeddings BioELMo (Jin et al, 2019) for word representations. Then we apply the ESIM model (Chen et al, 2016), where a biLSTM is used to encode the question and context/long answer, followed by an attentional local inference layer and a biLSTM inference composition layer.…”

Incorporating Domain Knowledge into Medical NLI using Knowledge Graphs

“…Accuracy fastText 68.7% GloVe 73.1% BioELM o (Jin et al, 2019) 78.2% ESIM w/K Romanov and Shivade (2018). Adding knowledge graph information to the base models showed an absolute improvement of 4.97% in case of fastText and 1.36% in case of GloVe.…”

“…Adding knowledge graph information to the base models showed an absolute improvement of 4.97% in case of fastText and 1.36% in case of GloVe. The baseline model utilizing BioELMo as base embeddings (Jin et al, 2019) showed an accuracy of 78.2%. On adding knowledge graph information, we were able to improve these results to 78.76% and on further addition of sentiment information, the accuracy rose to 79.04%…”

“…Following this stream of literature, Lee et al (2019) introduce BioBERT, which is a BERT model pretrained on English Wikipedia, BooksCorpus and fine-tuned on PubMed (7.8B tokens in total) corpus, PMC full-text articles. Jin et al (2019) propose BioELMo which is a domain-specific version of ELMo trained on 10M PubMed abstracts, and attempt to solve medical NLI problem with these domain specific embeddings, leading to state-ofthe-art performance. These two attempts show a direction towards solving medical NLI problem where the pretrained embeddings are fine-tuned on medical corpus and are used in the state-of-the-art NLI architecture.…”

“…Additionally, we also incorporate the sentiment information for medical concepts given by MetaMap (Aronson and Lang, 2010) leading to further improvement of the performance. Note that, as state-of-theart baselines, we use the models proposed by Jin et al (2019) and . Since, both of these studies consider ESIM as the core NLI model which makes it more convenient for us to incorporate extra domain knowledge and to have a fair performance comparison with these state-ofthe-art models.…”