exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources

Abstract

We introduce exBERT, a training method to extend BERT pre-trained models from a general domain to a new pre-trained model for a specific domain with a new additive vocabulary under constrained training resources (i.e., constrained computation and data). exBERT uses a small extension module to learn to adapt an augmenting embedding for the new domain in the context of the original BERT's embedding of a general vocabulary. The exBERT training method is novel in learning the new vocabulary and the extension module while keeping the weights of the original BERT model fixed, resulting in a substantial reduction in required training resources. We pre-train exBERT with biomedical articles from ClinicalKey and PubMed Central, and study its performance on biomedical downstream benchmark tasks using the MTL-Bioinformatics-2016 dataset. We demonstrate that exBERT consistently outperforms prior approaches when using limited corpus and pretraining computation resources.

Efficient Domain Adaptation of Language Models via Adaptive Tokenization

“…While we compared different novel token sequence embedding techniques, we did not study different ways of identifying subtoken sequences to add. Comparing AT to approaches such adding whole word tokens Tai et al (2020) would confirm our hypothesis that phrase-like token sequences are useful.…”

Developing a Clinical Language Model for Swedish: Continued Pretraining of Generic BERT with In-Domain Data

“…As shown by previous work, this may lead to further improvements in performance on the clinical tasks. Another approach is to manually add specific tokens to the vocabulary of a pretrained model, as explored by Tai et al (2020). An informed set of tokens could potentially be extracted by a new tokenizer specifically trained on the in-domain data, and in a later step, incorporate the set difference to the original tokenizer's vocabulary.…”

Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation

“…Moreover, SciBERT (Beltagy et al, 2019) found that in-domain vocabulary is helpful but not significant while we attribute it to the inefficiency of implicit learning of in-domain vocabulary. To represent OOV words in multilingual settings, the mixture mapping method (Wang et al, 2019) utilized a mixture of English subwords embedding, but it has been shown useless for domain-specific words by Tai et al (2020). ExBERT (Tai et al, 2020) applied an extension module to adapt an augmenting embedding for the in-domain vocabulary but it still needs large continuous pre-training.…”

“…To represent OOV words in multilingual settings, the mixture mapping method (Wang et al, 2019) utilized a mixture of English subwords embedding, but it has been shown useless for domain-specific words by Tai et al (2020). ExBERT (Tai et al, 2020) applied an extension module to adapt an augmenting embedding for the in-domain vocabulary but it still needs large continuous pre-training. Similar to our work, they highlight the importance of the domain-specific words but all of these work neither explore the understanding of performance drop during a domain shift nor examine the importance of multi-grained information.…”