Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning

Abstract

Recent prompt-based approaches allow pretrained language models to achieve strong performances on few-shot finetuning by reformulating downstream tasks as a language modeling problem. In this work, we demonstrate that, despite its advantages on low data regimes, finetuned prompt-based models for sentence pair classification tasks still suffer from a common pitfall of adopting inference heuristics based on lexical overlap, e.g., models incorrectly assuming a sentence pair is of the same meaning because they consist of the same set of words. Interestingly, we find that this particular inference heuristic is significantly less present in the zero-shot evaluation of the prompt-based model, indicating how finetuning can be destructive to useful knowledge learned during the pretraining. We then show that adding a regularization that preserves pretraining weights is effective in mitigating this destructive tendency of few-shot finetuning. Our evaluation on three datasets demonstrates promising improvements on the three corresponding challenge datasets used to diagnose the inference heuristics. 1

Confounder balancing in adversarial domain adaptation for pre-trained large models fine-tuning

“…2. The prompt tuning (Prompt-T) for LLMs [49,50] does not fine-tune a model on a labeled dataset for a specific task. Instead, they instead ReWeighting [53] involves training a naive model to predict based on dataset biases and another, robust model in ensemble with it, encouraging the latter to focus on more generalizable data patterns.…”

Asking Questions about Scientific Articles—Identifying Large N Studies with LLMs

“…However, prompt-based methods assigned to LLMs have been shown to be limited in Named Entity Recognition (NER) tasks. Problems with zero-shot experiments and the lack of robust prompts to obtain the necessary information have been observed [15][16][17][18][19][20][21]. One solution was to develop strategies for automatically converting the problems to Question Answering (QA) models, which improved the zero-shot capability for NER [14].…”

A Semi-Supervised Approach for Aspect Category Detection and Aspect Term Extraction from Opinionated Text

“…In recent years, many state-of-the-art Transfer Learning methods have been proposed along with zero-shot learning [32], XLM-R [33], BERT [34], etc., in the hybrid domain under ABSA. Moreover, prompt-based approaches like Few Shots or Zero Shot have recently been applied in ABSA [35], whereas prompts are given to the pre-trained model to complete the task. The usage of pre-trained models for Zero-Shot Learning (ZSL) has significantly improved ABSA due to its ability to understand linguistic features and possess better text encoders [36].…”

“…Similarly, ACSA in train XML file contains more than 3000 sentences with eight aspect categories (e.g., 'staff', 'menu', 'food', 'price', 'service', 'miscellaneous', 'ambiance', 'place') with their polarities. Sample data capture of ATSA and ACSA are given below: //Aspect Category Detection <sentence> <text>I like the smaller portion size for dinner.< \text> <aspectCategories> <aspectCategory category="miscellaneous" polarity="negative" \> <aspectCategory category="food" polarity="neutral" \> < \aspectCategories> < \sentence> //Aspect Term Extraction <sentence> <text>Food is pretty good but the service is horrific.< \text> <aspectTerms> <aspectTerm from="0" polarity="positive" term="Food" to="4" \> <aspectTerm from="28" polarity="negative" term="service" to=" 35…”