Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension

Xu, Ying; Zhong, Xu; Yepes, Antonio Jose Jimeno; Lau, Jey Han

doi:10.1109/ijcnn48605.2020.9206891

Cited by 32 publications

(16 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This may be due to a local minima problem or the catastrophic forgetting. As shown by Xu et al, catastrophic forgetting can happen during fine-tuning step, by overwriting previous knowledge of the model with new distinct knowledge, leading to a loss of information on lower layers (Xu et al, 2019). This may have occurred since the linguistic characteristics of clinical texts are very different from the Portuguese corpus used during pre-training phase of Portuguese BERT.…”

Section: Effect Of Languagementioning

confidence: 99%

BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

Schneider¹,

Souza²,

Knafou³

et al. 2020

Proceedings of the 3rd Clinical Natural Language Processing Workshop

View full text Add to dashboard Cite

With the growing number of electronic health record data, clinical NLP tasks have become increasingly relevant to unlock valuable information from unstructured clinical text. Although the performance of downstream NLP tasks, such as named-entity recognition (NER), in English corpus has recently improved by contextualised language models, less research is available for clinical texts in low resource languages. Our goal is to assess a deep contextual embedding model for Portuguese, so called BioBERTpt, to support clinical and biomedical NER. We transfer learned information encoded in a multilingual-BERT model to a corpora of clinical narratives and biomedical-scientific papers in Brazilian Portuguese. To evaluate the performance of BioBERTpt, we ran NER experiments on two annotated corpora containing clinical narratives and compared the results with existing BERT models. Our in-domain model outperformed the baseline model in F1-score by 2.72%, achieving higher performance in 11 out of 13 assessed entities. We demonstrate that enriching contextual embedding models with domain literature can play an important role in improving performance for specific NLP tasks. The transfer learning process enhanced the Portuguese biomedical NER model by reducing the necessity of labeled data and the demand for retraining a whole new model.

show abstract

Section: Effect Of Languagementioning

confidence: 99%

BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

Schneider¹,

Souza²,

Knafou³

et al. 2020

Proceedings of the 3rd Clinical Natural Language Processing Workshop

View full text Add to dashboard Cite

show abstract

“…For the source performance, we see a substantial drop (20-40 accuracy points) after cross-lingual transfer (e.g. "XLMR+source" vs. "MF-XLMR+ST"), implying there is catastrophic forgetting [9,12,32,35] -the phenomenon where adapted neural models "forget" and perform poorly in the original domain/task. When we incorporate gold labels in the source domain in the self-training loop ("MF-multiBERT+ST+GL" or "MF-XLMR+ST+GL"), we found a surprising observation: not only was catastrophic forgetting overcame, but the source performance actually surpasses some supervised monolingual models, e.g.…”

Section: Resultsmentioning

confidence: 92%

Rumour Detection via Zero-shot Cross-lingual Transfer Learning

Lin¹,

Zhang²,

Lau³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Most rumour detection models for social media are designed for one specific language (mostly English). There are over 40 languages on Twitter and most languages lack annotated resources to build rumour detection models. In this paper we propose a zero-shot cross-lingual transfer learning framework that can adapt a rumour detection model trained for a source language to another target language. Our framework utilises pretrained multilingual language models (e.g. multilingual BERT) and a self-training loop to iteratively bootstrap the creation of "silver labels" in the target language to adapt the model from the source language to the target language. We evaluate our methodology on English and Chinese rumour datasets and demonstrate that our model substantially outperforms competitive benchmarks in both source and target language rumour detection.

show abstract

“…Since such changes in word probability are likely to impact the decoder and joint networks more than the encoder, a regularisation for the decoder and joint networks may also be required during fine-tuning. We experiment with EWC [17,20] for this purpose, with its loss function formulated as:…”

Section: Elastic Weight Consolidationmentioning

confidence: 99%

Using Synthetic Audio to Improve the Recognition of Out-of-Vocabulary Words in End-to-End Asr Systems

Zheng

Liu

Gunceler

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Today, many state-of-the-art automatic speech recognition (ASR) systems apply all-neural models that map audio to word sequences trained end-to-end along one global optimisation criterion in a fully data driven fashion. These models allow high precision ASR for domains and words represented in the training material but have difficulties recognising words that are rarely or not at all represented during training, i.e. trending words and new named entities. In this paper, we use a text-to-speech (TTS) engine to provide synthetic audio for out-of-vocabulary (OOV) words. We aim to boost the recognition accuracy of a recurrent neural network transducer (RNN-T) on OOV words by using the extra audio-text pairs, while maintaining the performance on the non-OOV words. Different regularisation techniques are explored and the best performance is achieved by fine-tuning the RNN-T on both original training data and extra synthetic data with elastic weight consolidation (EWC) applied on the encoder. This yields a 57% relative word error rate (WER) reduction on utterances containing OOV words without any degradation on the whole test set.

show abstract

Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension

Cited by 32 publications

References 13 publications

BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

Rumour Detection via Zero-shot Cross-lingual Transfer Learning

Using Synthetic Audio to Improve the Recognition of Out-of-Vocabulary Words in End-to-End Asr Systems

Contact Info

Product

Resources

About