Knowledge distillation for BERT unsupervised domain adaptation

Ryu, Minho; Lee, Kichun

doi:10.1007/s10115-022-01736-y

Cited by 21 publications

(14 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given training data from a labeled source domain D train S and an unlabeled target domain D train T , our approach for DA in hate-speech involves 2 steps: (i) extraction of source-specific terms and (ii) reducing the importance of these terms. Our setting is similar to Ben-David et al (2020) and Ryu and Lee (2020).…”

Section: Proposed Approachmentioning

confidence: 99%

“…Our method, through penalization of these terms, automatically enforces the source domain classifier to focus on domain-invariant content. Compared to approaches transforming highdimensional intermediate representations to reduce the domain discrepancy, such as domain adversarial learning (Ryu and Lee, 2020;Tzeng et al, 2017), our approach makes the adaptation more explainable, while improving the overall cross-domain performance compared to prior-approaches.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Domain Classification-based Source-specific Term Penalization for Domain Adaptation in Hate-speech Detection

Bose¹,

Αλέτρας²,

Illina³

et al. 2022

Preprint

View full text Add to dashboard Cite

Warning: this paper contains content that may be offensive and distressing.State-of-the-art approaches for hate-speech detection usually exhibit poor performance in out-of-domain settings. This occurs, typically, due to classifiers overemphasizing sourcespecific information that negatively impacts its domain invariance. Prior work has attempted to penalize terms related to hatespeech from manually curated lists using feature attribution methods, which quantify the importance assigned to input terms by the classifier when making a prediction. We, instead, propose a domain adaptation approach that automatically extracts and penalizes source-specific terms using a domain classifier, which learns to differentiate between domains, and feature-attribution scores for hatespeech classes, yielding consistent improvements in cross-domain evaluation.

show abstract

Section: Proposed Approachmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Domain Classification-based Source-specific Term Penalization for Domain Adaptation in Hate-speech Detection

Bose¹,

Αλέτρας²,

Illina³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Studies in [22], [47] found that a catastrophic forgetting problem occurs when the ADDA framework is applied to the BERT model. [47] propose a look-ahead optimization strategy to accommodate the adversarial domain discrimination loss and the task-specific classification loss when optimizing BERT representations.…”

Section: B Adversarial Learningmentioning

confidence: 99%

“…[47] propose a look-ahead optimization strategy to accommodate the adversarial domain discrimination loss and the task-specific classification loss when optimizing BERT representations. [22] propose to use knowledge distillation [119] to distill knowledge from source encoder to target encoder, thereby regularizing ADDA for unsupervised domain adaptation of BERT. Adversarial Robustness and Consistency Training.…”

Section: B Adversarial Learningmentioning

confidence: 99%

“…To conduct successful knowledge transfer, a range of domain adaptation methods have been proposed based on different assumptions of the given data and the training setting. However, traditional domain adaptation methods that work with shallow neural networks or non-pretrained language models can be unfavorable to PLMs [22]. For example, [23], [24] find that domain adversarial training on top of BERT is unstable and has little effect on cross-domain performance, suggesting that proper design of domain adaptation methods is necessary for PLMs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

Xu¹,

Han²

2022

Preprint

View full text Add to dashboard Cite

Recent advances in NLP are brought by a range of large-scale pretrained language models (PLMs). These PLMs have brought significant performance gains for a range of NLP tasks, circumventing the need to customize complex designs for specific tasks. However, most current work focus on finetuning PLMs on a domain-specific datasets, ignoring the fact that the domain gap can lead to overfitting and even performance drop. Therefore, it is practically important to find an appropriate method to effectively adapt PLMs to a target domain of interest. Recently, a range of methods have been proposed to achieve this purpose. Early surveys on domain adaptation are not suitable for PLMs due to the sophisticated behavior exhibited by PLMs from traditional models trained from scratch and that domain adaptation of PLMs need to be redesigned to take effect. This paper aims to provide a survey on these newly proposed methods and shed light in how to apply traditional machine learning methods to newly evolved and future technologies. By examining the issues of deploying PLMs for downstream tasks, we propose a taxonomy of domain adaptation approaches from a machine learning system view, covering methods for input augmentation, model optimization and personalization. We discuss and compare those methods and suggest promising future research directions.

show abstract

A Contrastive Self-distillation BERT with Kernel Alignment-Based Inference

Yuan

Cao

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Knowledge distillation for BERT unsupervised domain adaptation

Cited by 21 publications

References 12 publications

Domain Classification-based Source-specific Term Penalization for Domain Adaptation in Hate-speech Detection

Domain Classification-based Source-specific Term Penalization for Domain Adaptation in Hate-speech Detection

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

A Contrastive Self-distillation BERT with Kernel Alignment-Based Inference

Contact Info

Product

Resources

About