Towards Distribution-shift Robust Text Classification of Emotional Content

Abstract

Supervised models based on Transformers have been shown to achieve impressive performances in many natural language processing tasks. However, besides requiring a large amount of costly manually annotated data, supervised models tend to adapt to the characteristics of the training dataset, which are usually created ad-hoc and whose data distribution often differs from the one in real applications, showing significant performance degradation in real-world scenarios. We perform an extensive assessment of the out-of-distribution performances of supervised models for classification in the emotion and hate-speech detection tasks and show that NLI-based zeroshot models often outperform them, making task-specific annotation useless when the characteristics of final-user data are not known in advance. To benefit from both supervised and zero-shot approaches, we propose to finetune an NLI-based model on the task-specific dataset. The resulting model often outperforms all available supervised models both in distribution and out of distribution, with only a few thousand training samples.

Publish or Hold? Automatic Comment Moderation in Luxembourgish News Articles

“…We also experimented with several state-of-theart text-to-text transformers, which treat all tasks as text generation problems. These transformers have provided excellent results in text classification tasks (Bulla et al, 2023;Sabry et al, 2022;Ni et al, 2022). They do not rely on a classification layer (Raffel et al, 2020) and have a flexible input-output format.…”