Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension

Xu, Yeshou; Zhong, Xing; Yepes, Antonio Jose Jimeno; Lau, Jey Han

doi:10.48550/arxiv.1911.00202

Cited by 3 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In future work, we intend to explore various methods to improve the performance of T+DAPT by remedying catastrophic forgetting and maximizing knowledge transfer. For this we hope to emulate the regularization used by Xu et al (2020) and implement multi-task learning and continual learning methods like AdapterNet (Hazan et al, 2018). In order to improve the transferability of learned features, we will explore different auxiliary tasks such as NLI and sentiment analysis in addition to few-shot learning approaches.…”

Section: Discussionmentioning

confidence: 99%

“…We train our own DAPT baselines on the Movies and COVID-19 domains. Xu et al (2020) explore methods to reduce catastrophic forgetting during language model fine-tuning. They apply topic modeling on the MS MARCO dataset (Bajaj et al, 2018) to generate 6 narrow domain-specific data sets, from which we use BioQA and MoviesQA as domain-specific reading comprehension benchmarks.…”

Section: Related Workmentioning

confidence: 99%

“…We explore the performance of this approach in the Movies, News, Biomedical, and COVID-19 domains. Specifically, our target domain-specific MRC tasks are MoviesQA (Xu et al, 2020), NewsQA (Trischler et al, 2017), BioQA (Xu et al, 2020), and CovidQA…”

Section: Datasetsmentioning

confidence: 99%

See 2 more Smart Citations

Task Transfer and Domain Adaptation for Zero-Shot Question Answering

Peng¹,

Sheng²,

Shimshoni³

et al. 2022

Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing

View full text Add to dashboard Cite

Pretrained language models have shown success in various areas of natural language processing, including reading comprehension tasks. However, when applying machine learning methods to new domains, labeled data may not always be available. To address this, we use supervised pretraining on source-domain data to reduce sample complexity on domainspecific downstream tasks. We evaluate zeroshot performance on domain-specific reading comprehension tasks by combining task transfer with domain adaptation to fine-tune a pretrained model with no labelled data from the target task. Our approach outperforms Domain-Adaptive Pretraining on downstream domain-specific reading comprehension tasks in 3 out of 4 domains.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Task Transfer and Domain Adaptation for Zero-Shot Question Answering

Peng¹,

Sheng²,

Shimshoni³

et al. 2022

Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…To deal with this issue, He et al (2019) proposed to replay pre-train data during fine-tuning heavily, and Chen et al (2020) proposed an improved Adam optimizer to recall knowledge captured during pretraining. The catastrophic forgetting issue is also noticed in domain adaptation setups for neural machine translation (Saunders et al, 2019;Thompson et al, 2019;Varis and Bojar, 2019) and the reading comprehension task (Xu et al, 2019). Lee (2017) firstly studied the continual learning setting for dialog state tracking in task-oriented dialog systems.…”

Section: Introductionmentioning

confidence: 99%

Continual Learning for Natural Language Generation in Task-oriented Dialog Systems

Mi¹,

Chen²,

Zhao³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Natural language generation (NLG) is an essential component of task-oriented dialog systems. Despite the recent success of neural approaches for NLG, they are typically developed in an offline manner for particular domains. To better fit real-life applications where new data come in a stream, we study NLG in a "continual learning" setting to expand its knowledge to new domains or functionalities incrementally. The major challenge towards this goal is catastrophic forgetting, meaning that a continually trained model tends to forget the knowledge it has learned before. To this end, we propose a method called ARPER (Adaptively Regularized Prioritized Exemplar Replay) by replaying prioritized historical exemplars, together with an adaptive regularization technique based on Elastic Weight Consolidation. Extensive experiments to continually learn new domains and intents are conducted on MultiWoZ-2.0 to benchmark ARPER with a wide range of techniques. Empirical results demonstrate that ARPER significantly outperforms other methods by effectively mitigating the detrimental catastrophic forgetting issue.

show abstract