Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks

Abstract

Pre-trained transformer models have shown enormous success in improving performance on several downstream tasks. However, fine-tuning on a new task still requires large amounts of taskspecific labeled data to achieve good performance. We consider this problem of learning to generalize to new tasks with a few examples as a meta-learning problem. While meta-learning has shown tremendous progress in recent years, its application is still limited to simulated problems or problems with limited diversity across tasks. We develop a novel method, LEOPARD, which enables optimization-based meta-learning across tasks with different number of classes, and evaluate different methods on generalization to diverse NLP classification tasks. LEOP-ARD is trained with the state-of-the-art transformer architecture and shows better generalization to tasks not seen at all during training, with as few as 4 examples per label. Across 17 NLP tasks, including diverse domains of entity typing, natural language inference, sentiment analysis, and several other text classification tasks, we show that LEOPARD learns better initial parameters for few-shot learning than self-supervised pre-training or multi-task training, outperforming many strong baselines, for example, yielding 14.6% average relative gain in accuracy on unseen tasks with only 4 examples per label.

Don’t Miss the Labels: Label-semantic Augmented Meta-Learner for Few-Shot Text Classification

“…Pre-trained language models have also been applied to few-shot text classification. LEOPARD (Bansal et al, 2020) uses BERT (Devlin et al, 2019) with optimization-based metalearning framework to achieve good performance on diverse NLP classification tasks. More recently, GPT-3 (Brown et al, 2020) shows that the language model itself can be used to perform few-shot text classification without using meta-learning.…”

DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference

“…Since these applications come with well-defined task distributions, they do not have the same overfitting challenges. On the other hand, many works deal with few-shot adaptation in settings with no clear task distribution (Dou et al, 2019;Bansal et al, 2020a) but do not address meta-overfitting, and thus are complementary to our work.…”

“…On GLUE-SciTail, we compare against SMLMT (Bansal et al, 2020b) and find that MAML-DRECA improves over it by 1.5 accuracy points. However, we note that the confidence intervals of these approaches overlap, and also that (Bansal et al, 2020a) consider the entire GLUE data to train the meta-learner whereas we only consider NLI datasets within GLUE. Table 3: Results on NLI few-shot learning.…”

Meta Learning and Its Applications to Natural Language Processing

“…Text classification has a vast spectrum of applications, such as sentiment classification and intent classification. The meta-learning algorithms developed for image classification can be applied to text classification with slight modification to incorporate domain knowledge in each application Tan et al, 2019;Geng et al, 2019;Dou et al, 2019;Bansal et al, 2019).…”