An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training

Arumae, Kristjan; Sun, Qing; Bhatia, Parminder

doi:10.18653/v1/2020.emnlp-main.394

Cited by 11 publications

(9 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results are shown in Table 5. Among the sentence-wise and the token-wise classification variants, the SQP(EWC+LRD) gives the best results considering exact match, ROUGE-L F1 and S+WMS scores, while the SQP(SLR) and the SQP(FT RB) variants perform the poorest among the lot, which is consistent with the results in Arumae et al (2020). It only produces short answers, hence have a high precision but is poor on all other counts.…”

Section: Resultssupporting

confidence: 82%

“…(a). Using a learning rate that linearly decreases by a constant factor (LRD) from one layer to the next, with the outermost language modeling head layer having the maximum learning rate, as in Arumae et al (2020). This enforces a constraint that outer layers adapt more to the E-Manual domain, while the inner layers' weights do not change much, thus restricting them to retain the knowledge of the generic domain primarily.…”

Section: Pre-training On the E-manuals Corpusmentioning

confidence: 99%

“…(b). Using elastic weight consolidation (EWC) (Kirkpatrick et al, 2017;Arumae et al, 2020) to mitigate catastrophic forgetting while switching from the generic domain on which original "RoBERTa" was pre-trained to the domain of E-Manuals. A batch size of 64 is used.…”

Section: Pre-training On the E-manuals Corpusmentioning

confidence: 99%

See 2 more Smart Citations

Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework

Nandy

Sharma²,

Maddhashiya³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

Answering questions asked from instructional corpora such as E-manuals, recipe books, etc., has been far less studied than open-domain factoid context-based question answering. This can be primarily attributed to the absence of standard benchmark datasets. In this paper we meticulously create a large amount of data connected with E-manuals and develop suitable algorithm to exploit it. We collect E-Manual Corpus, a huge corpus of 307,957 E-manuals and pretrain RoBERTa on this large corpus. We create various benchmark QA datasets which include question answer pairs curated by experts based upon two E-manuals, real user questions from Community Question Answering Forum pertaining to E-manuals etc.We introduce EMQAP (E-Manual Question Answering Pipeline) that answers questions pertaining to electronics devices. Built upon the pretrained RoBERTa, it harbors a supervised multi-task learning framework which efficiently performs the dual tasks of identifying the section in the E-manual where the answer can be found and the exact answer span within that section.For E-Manual annotated question-answer pairs, we show an improvement of about 40% in ROUGE-L F1 scores over the most competitive baseline. We perform a detailed ablation study and establish the versatility of EMQAP across different circumstances. The code and datasets are shared at https://github.com/abhi1nandy2/ EMNLP-2021-Findings, and the corresponding project website is https://sites. google.com/view/emanualqa/home.

show abstract

Section: Resultssupporting

confidence: 82%

Section: Pre-training On the E-manuals Corpusmentioning

confidence: 99%

See 1 more Smart Citation

Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework

Nandy

Sharma²,

Maddhashiya³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

show abstract

“…A number of recent works study LM adaptation to a new domain that is different from the original pretraining domain. They show performance gains on downstream tasks in the new domain (Gururangan et al, 2020;Yao et al, 2021), as well as retaining knowledge learned in general domain when certain regularizations are applied (Arumae et al, 2020). However, continual (sequential) pretraining over multiple distinct corpora is less studiedit is unclear how the LM can continually acquire and accumulate knowledge from different corpora to benefit downstream tasks in a new domain or on more recent data, and whether the LM can retain the knowledge learned from earlier corpora to preserve decent performance on seen domains.…”

Section: Introductionmentioning

confidence: 99%

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

Jin¹,

Zhang²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Pretrained language models (PTLMs) are typically learned over a large, static corpus and further fine-tuned for various downstream tasks. However, when deployed in the real world, a PTLM-based model must deal with data from a new domain that deviates from what the PTLM was initially trained on, or newly emerged data that contains out-of-distribution information. In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologicallyordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithms, and keep track of the downstream task performance (after fine-tuning) to analyze its ability of acquiring new knowledge and preserving learned knowledge. Our experiments show continual learning algorithms improve knowledge preservation, with logit distillation being the most effective approach. We further show that continual pretraining improves generalization when training and testing data of downstream tasks are drawn from different time steps, but do not improve when they are from the same time steps. We believe our problem formulation, methods, and analysis will inspire future studies towards continual pretraining of language models.

show abstract

“…CF mitigation is particularly important in clinical NLP given that many clinical datasets are quite different from generic domains and from each other. Arumae et al [2020] explore CF in language modeling when transferring between the generic, clinical, and biomedical domains and compare learning rate control, experience replay, and EWC.…”

Section: Discussionmentioning

confidence: 99%

Kronecker Factorization for Preventing Catastrophic Forgetting in Large-scale Medical Entity Linking

McInerney¹,

Kong²,

Arumae³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Multi-task learning is useful in NLP because it is often practically desirable to have a single model that works across a range of tasks. In the medical domain, sequential training on tasks may sometimes be the only way to train models, either because access to the original (potentially sensitive) data is no longer available, or simply owing to the computational costs inherent to joint retraining. A major issue inherent to sequential learning, however, is catastrophic forgetting, i.e., a substantial drop in accuracy on prior tasks when a model is updated for a new task. Elastic Weight Consolidation is a recently proposed method to address this issue, but scaling this approach to the modern large models used in practice requires making strong independence assumptions about model parameters, limiting its effectiveness. In this work, we apply Kronecker Factorization-a recent approach that relaxes independence assumptions-to prevent catastrophic forgetting in convolutional and Transformer-based neural networks at scale. We show the effectiveness of this technique on the important and illustrative task of medical entity linking across three datasets, demonstrating the capability of the technique to be used to make efficient updates to existing methods as new medical data becomes available. On average, the proposed method reduces catastrophic forgetting by 51% when using a BERT-based model, compared to a 27% reduction using standard Elastic Weight Consolidation, while maintaining spatial complexity proportional to the number of model parameters.

show abstract

An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training

Cited by 11 publications

References 29 publications

Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework

Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

Kronecker Factorization for Preventing Catastrophic Forgetting in Large-scale Medical Entity Linking

Contact Info

Product

Resources

About