Improved Text Classification via Contrastive Adversarial Training

Abstract

We propose a simple and general method to regularize the fine-tuning of Transformer-based encoders for text classification tasks. Specifically, during fine-tuning we generate adversarial examples by perturbing the word embedding matrix of the model and perform contrastive learning on clean and adversarial examples in order to teach the model to learn noise-invariant representations. By training on both clean and adversarial examples along with the additional contrastive objective, we observe consistent improvement over standard fine-tuning on clean examples. On several GLUE benchmark tasks, our fine-tuned Bert_Large model outperforms Bert_Large baseline by 1.7% on average, and our fine-tuned Roberta_Large improves over Roberta_Large baseline by 1.3%. We additionally validate our method in different domains using three intent classification datasets, where our fine-tuned Roberta_Large outperforms Roberta_Large baseline by 1-2% on average. For the challenging low-resource scenario, we train our system using half of the training data (per intent) in each of the three intent classification datasets, and achieve similar performance compared to the baseline trained with full training data.

ANN: adversarial news net for robust fake news classification

“…A novel technique for regularizing the fine-tuning of Transformer-based encoders for text classification problems is provided in Ref. 18 . The model’s word embedding matrix is perturbed to provide adversarial examples, and contrastive learning is used to train the model to learn noise-invariant representation using clean and adversarial examples.…”

“…AraBERTv2 achieved the highest accuracy, precision, and f1-score of 97% on the description dataset among all other transformer architectures It is only restricted to Arabic news content 16 2022 Pre-trained BERT model Detect fake news in a region-based distributed approach FakeNewsNet dataset Precision, recall, and F1-score The model achieved an accuracy of 91% The distributed framework further needs to be optimized in the mobile crowdsensing environment 17 2021 A hybrid model combining RNN with Bidirectional GRU and SVM Identify real and fake news FakeNewsNet dataset Accuracy, precision, recall, and F1-score Suggested methodology performed better than cutting-edge techniques The limitation of Support Vector Machines (SVM) is that their performance depends on the feature vector's size. In this case, the minimum size of the feature vector was restricted to 512 units, which is the output of the GRUs 18 2022 Perturbation of word embedding matrix and contrastive learning using transformers such as BERT and RoBERTa GLUE benchmark tasks and three intent classification datasets Accuracy The method demonstrates an improvement of 1.7% on average over BERTLarge and 1.3% over RoBERTaLarge. On intent classification tasks, the fine-tuned RoBERTaLarge outperforms the RoBERTaLarge baseline by 1% on the entire test sets and 2% on the more challenging test sets Regularizing Transformer-based encoders for text classification problems Modest perturbations to input vector entries may not be appropriate for sparse high-dimensional inputs 19 2022 CRAL (consistent regularization for adaptation learning) and VAT (virtual adversarial training) with entropy minimization Two MDTC (multi-domain text classification) benchmarks Accuracy 88% and 90% on both datasets Adversarial training for specific domain adaptation Accuracy is compromised in an unseen domain …”

Enhancing Machine-Generated Text Detection: Adversarial Fine-Tuning of Pre-Trained Language Models

“…In particular, because of the convergence of models such as BERT [18], XLNet [34], RoBERTa [13], T5 [15], and ELECTRA [35], research has been conducted to apply AT to the fine-tuning process of pre-trained language models. AT has been shown to improve the performance of text classification tasks [36], and AT is also effective for fine-tuning and pre-training language models [37]- [39]. It has been empirically demonstrated that AT is effective when applied to a BERT model [40].…”

ANN: Adversarial News Net

“…Their research shows that Roberta Large also performs 1-2% better than the Roberta Large baseline. In reference 17 , the authors propose a novel method called CRAL (Consistent Regularization for Adaptation Learning) for performing domain adaptation. The approach involves creating two distinct shared latent spaces, conducting domain alignment for each space, and penalizing any inconsistencies between the two alignments in terms of predictions for unlabeled data.…”