Making Attention Mechanisms More Robust and Interpretable with Virtual Adversarial Training for Semi-Supervised Text Classification

Kitada, Shunsuke; Iyatomi, Hitoshi

doi:10.48550/arxiv.2104.08763

Cited by 3 publications

(2 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[28] perturbed word embeddings instead of original input text using FGSM for NLP task. Some of the recent works [20,21,44] added perturbations to the attention mechanism of transformer methods using FGSM. [26] used multi-step FGSM to generate adversarial examples that proved more effective at the cost of computational overhead.…”

Section: Adversarial Trainingmentioning

confidence: 99%

A Novel Approach to Train Diverse Types of Language Models for Health Mention Classification of Tweets

Khan¹,

Razzak²,

Dengel³

et al. 2022

Preprint

View full text Add to dashboard Cite

Health mention classification deals with the disease detection in a given text containing disease words. However, non-health and figurative use of disease words adds challenges to the task. Recently, adversarial training acting as a means of regularization has gained popularity in many NLP tasks. In this paper, we propose a novel approach to train language models for health mention classification of tweets that involves adversarial training. We generate adversarial examples by adding perturbation to the representations of transformer models for tweet examples at various levels using Gaussian noise. Further, we employ contrastive loss as an additional objective function. We evaluate the proposed method on the PHM2017 dataset extended version. Results show that our proposed approach improves the performance of classifier significantly over the baseline methods. Moreover, our analysis shows that adding noise at earlier layers improves models' performance whereas adding noise at intermediate layers deteriorates models' performance. Finally, adding noise towards the final layers performs better than the middle layers noise addition.

show abstract

Section: Adversarial Trainingmentioning

confidence: 99%

A Novel Approach to Train Diverse Types of Language Models for Health Mention Classification of Tweets

Khan¹,

Razzak²,

Dengel³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[43] applies AT to relation extraction. Recent works [21,22,47] propose to apply perturbations to the attention mechanism in Transformer-based encoders. Compared to single-step FGSM, [26] demonstrates the superior effectiveness of the multi-step approach to generate perturbed examples with projected gradient descent, which comes at a greater computational cost due to the inner loop that iteratively calculates the perturbations.…”

Section: Related Work 21 Adversarial Trainingmentioning

confidence: 99%

Improved Text Classification via Contrastive Adversarial Training

Lin¹,

Hang²,

Sil³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a simple and general method to regularize the fine-tuning of Transformer-based encoders for text classification tasks. Specifically, during finetuning we generate adversarial examples by perturbing the word embeddings of the model and perform contrastive learning on clean and adversarial examples in order to teach the model to learn noise-invariant representations. By training on both clean and adversarial examples along with the additional contrastive objective, we observe consistent improvement over standard fine-tuning on clean examples. On several GLUE benchmark tasks, our fine-tuned BERT Large model outperforms BERT Large baseline by 1.7% on average, and our fine-tuned RoBERTa Large improves over RoBERTa Large baseline by 1.3%. We additionally validate our method in different domains using three intent classification datasets, where our fine-tuned RoBERTa Large outperforms RoBERTa Large baseline by 1-2% on average.Preprint. Under review.

show abstract

Improved Text Classification via Contrastive Adversarial Training

Pan¹,

Hang²,

Sil³

et al. 2022

AAAI

View full text Add to dashboard Cite

We propose a simple and general method to regularize the fine-tuning of Transformer-based encoders for text classification tasks. Specifically, during fine-tuning we generate adversarial examples by perturbing the word embedding matrix of the model and perform contrastive learning on clean and adversarial examples in order to teach the model to learn noise-invariant representations. By training on both clean and adversarial examples along with the additional contrastive objective, we observe consistent improvement over standard fine-tuning on clean examples. On several GLUE benchmark tasks, our fine-tuned Bert_Large model outperforms Bert_Large baseline by 1.7% on average, and our fine-tuned Roberta_Large improves over Roberta_Large baseline by 1.3%. We additionally validate our method in different domains using three intent classification datasets, where our fine-tuned Roberta_Large outperforms Roberta_Large baseline by 1-2% on average. For the challenging low-resource scenario, we train our system using half of the training data (per intent) in each of the three intent classification datasets, and achieve similar performance compared to the baseline trained with full training data.

show abstract

Making Attention Mechanisms More Robust and Interpretable with Virtual Adversarial Training for Semi-Supervised Text Classification

Cited by 3 publications

References 27 publications

A Novel Approach to Train Diverse Types of Language Models for Health Mention Classification of Tweets

A Novel Approach to Train Diverse Types of Language Models for Health Mention Classification of Tweets

Improved Text Classification via Contrastive Adversarial Training

Improved Text Classification via Contrastive Adversarial Training

Contact Info

Product

Resources

About