Zero-Shot Topic Labeling for Hazard Classification

Abstract

Topic classification is the task of mapping text onto a set of meaningful labels known beforehand. This scenario is very common both in academia and industry whenever there is the need of categorizing a big corpus of documents according to set custom labels. The standard supervised approach, however, requires thousands of documents to be manually labelled, and additional effort every time the label taxonomy changes. To obviate these downsides, we investigated the application of a zero-shot approach to topic classification. In this setting, a subset of these topics, or even all of them, is not seen at training time, challenging the model to classify corresponding examples using additional information. We first show how zero-shot classification can perform the topic-classification task without any supervision. Secondly, we build a novel hazard-detection dataset by manually selecting tweets gathered by LINKS Foundation for this task, where we demonstrate the effectivenes of our cost-free method on a real-world problem. The idea is to leverage a pre-trained text-embedder (MPNet) to map both text and topics into the same semantic vector space where they can be compared. We demonstrate that these semantic spaces are better aligned when their dimension is reduced, keeping only the most useful information. We investigated three different dimensionality reduction techniques, namely, linear projection, autoencoding and PCA. Using the macro F1-score as the standard metric, it was found that PCA is the best performing technique, recording improvements for each dataset in comparison with the performance on the baseline.

Data Sorting Influence on Short Text Manual Labeling Quality for Hierarchical Classification

“…Zero-shot learning presents a promising approach to the challenge of automatic labeling in short text datasets [40]. This technique, which allows a machine learning model to classify data accurately that it has never seen before, is particularly well suited to short text data's dynamic and diverse nature [41,42]. One of the most powerful tools for zero-shot learning in NLP is the Bidirectional and Auto-Regressive Transformers (BART) model from Hugging Face, specifically the 'bart-large-mnli' variant [43].…”