Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

Abstract

Zero-shot text classification (0SHOT-TC) is a challenging NLU problem to which little attention has been paid by the research community. 0SHOT-TC aims to associate an appropriate label with a piece of text, irrespective of the text domain and the aspect (e.g., topic, emotion, event, etc.) described by the label. And there are only a few articles studying 0SHOT-TC, all focusing only on topical categorization which, we argue, is just the tip of the iceberg in 0SHOT-TC. In addition, the chaotic experiments in literature make no uniform comparison, which blurs the progress.This work benchmarks the 0SHOT-TC problem by providing unified datasets, standardized evaluations, and state-of-the-art baselines. Our contributions include: i) The datasets we provide facilitate studying 0SHOT-TC relative to conceptually different and diverse aspects: the "topic" aspect includes "sports" and "politics" as labels; the "emotion" aspect includes "joy" and "anger"; the "situation" aspect includes "medical assistance" and "water shortage". ii) We extend the existing evaluation setup (labelpartially-unseen) -given a dataset, train on some labels, test on all labels -to include a more challenging yet realistic evaluation label-fully-unseen 0SHOT-TC (Chang et al., 2008), aiming at classifying text snippets without seeing task specific training data at all. iii) We unify the 0SHOT-TC of diverse aspects within a textual entailment formulation and study it this way. 1The plague in Mongolia, occurring last week, has caused more than a thousand isolation health,

Text Classification Using Label Names Only: A Language Model Self-Training Approach

“…Early attempts rely on distant supervision such as Wikipedia to interpret the label name semantics and derive documentconcept relevance via explicit semantic analysis (Gabrilovich and Markovitch, 2007). Since the classifier is learned purely from general knowledge without even requiring any unlabeled domainspecific data, these methods are called dataless classification (Chang et al, 2008;Song and Roth, 2014;Yin et al, 2019). Later, topic models (Chen et al, 2015;Li et al, 2016) are exploited for seedguided classification to learn seed word-aware topics by biasing the Dirichlet priors and to infer posterior document-topic assignment.…”

Learning from Task Descriptions

“…Recent work in NLP has shown significant progress in learning tasks from examples. Large pretrained language models have dramatically improved performance on standard benchmarks (Peters et al, 2018;Devlin et al, 2019;Raffel et al, 2019) and have shown promising results in zero shot prediction by leveraging their language understanding capabilities (Levy et al, 2017;Zhou et al, 2018;Yin et al, 2019).…”

Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start

“…This coreference example shows that transforming an NLP task as textual entailment may obtain surprising advantages. There are more NLP tasks that can fit the entailment framework easily, such as text classification (Yin et al, 2019), relation extraction, summarization, etc. However, we also need to admit that reformulating into entailment may also need to fight against new challenges.…”