Assessment of Massively Multilingual Sentiment Classifiers

Abstract

Models are increasing in size and complexity in the hunt for SOTA. But what if those 2% increase in performance does not make a difference in a production use case? Maybe benefits from a smaller, faster model outweigh those slight performance gains. Also, equally good performance across languages in multilingual tasks is more important than SOTA results on a single one. We present the biggest, unified, multilingual collection of sentiment analysis datasets. We use these to assess 11 models and 80 high-quality sentiment datasets (out of 342 raw datasets collected) in 27 languages and included results on the internally annotated datasets. We deeply evaluate multiple setups, including fine-tuning transformer-based models for measuring performance. We compare results in numerous dimensions addressing the imbalance in both languages coverage and dataset sizes. Finally, we present some best practices for working with such a massive collection of datasets and models from a multilingual perspective.

The Paradox of Multilingual Emotion Detection

“…I selected three WASSA submissions from 2021 and 2022, namely those that included the highest number of languages: the papers of Rajda et al (2022), Bianchi et al (2022), and Lamprinidis et al (2021). The first one includes an assessment of sentiment analysis in 27 languages (I will refer to this work as MSA, standing for multilingual sentiment analysis); the second one presents XLM-EMO, a multilingual emotion detection model evaluated on 19 languages; and the last one presents Universal Joy, an emotion detection dataset including 18 languages.…”

“…The data: Both in MSA (Rajda et al, 2022) and Universal Joy (Lamprinidis et al, 2021), data is used that was originally written in the target languages. While existing sentiment datasets are used in MSA, Universal Joy is created by scraping Facebook posts based on the Facebook-specific feelings tags.…”

“…I believe that this positive trend is fueled by large language models like multilingual BERT (Devlin et al, 2019), which lend themselves perfectly to (zero or few-shot) transfer learning. Indeed, the high number of languages at WASSA 2021 and 2022 is largely due to just three papers that include many different languages: Lamprinidis et al (2021), Bianchi et al (2022) and Rajda et al (2022) respectively include 18, 19 and 27 languages, each of them performing experiments using multilingual models.…”