TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

Abstract

The experimental landscape in natural language processing for social media is too fragmented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domainspecific data. In this paper, we propose a new evaluation framework (TWEETEVAL) consisting of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pretrained generic language models, and continue training them on Twitter corpora.

Collective Emotions during the COVID-19 Outbreak

“…Additionally, we excluded common English positive words that have been found to be negatively correlated with well-being and happiness at the regional level, namely "love", "good", "LOL", "better", "well" and "like" (Jaidka et al, 2020). We test the robustness of the LIWC analysis by comparing with the results of a machine learning model for emotion categorization (Barbieri et al, 2020) based on a RoBERTa (Liu et al, 2019) supervised classifier trained with the annotated tweets from the SemEval dataset (Mohammad et al, 2018).…”

“…For two example countries (USA, Spain), we checked the robustness of our dictionary-based emotion measures by comparing it with measures based on a machine learning model for emotion categorization. For this, we finetuned the pretrained masked language model from the TweetEval Benchmark (Barbieri et al, 2020) -a model based on RoBERTa-base (Liu et al, 2019) -on English and Spanish tweets from the SemEval dataset (Mohammad et al, 2018). We then correlated the percentage of emotional tweets based on dictionary-based (anxiety, anger, sadness, positive LIWC) and machine-learning based emotion labels (fear, anger, sadness and joy + optimism + love predicted by RoBERTa).…”

Multi-input Recurrent Independent Mechanisms for leveraging knowledge sources: Case studies on sentiment analysis and health text mining

“…Comparison with SOTA The SST-2, SE17-4A, SM20-5 tasks have been deployed on GLUE, TweetEval, and Codalab benchmarks respectively, therefore, the state of the art (SOTA) results are available. Current SOTA performances on SST-2 are obtained by (Sun et al, 2019) and (Raffel et al, 2020) (tied), SOTA for SE17-4A is reported by (Barbieri et al, 2020), and SOTA for SM20-5 is reported by (Bai and Zhou, 2020) as shown in Figure 10.…”

“…The tweets are classified as Negative, Neutral, and Positive. The performance for this task is measured by the macro-average of recall scores for positive, negative, and neutral classes and evaluated by the TweetEval benchmark (Barbieri et al, 2020) 5 .…”

Fighting the COVID-19 Infodemic with a Holistic BERT Ensemble

“…TWEETEVAL (Barbieri et al, 2020) is a pretrained RoBERTa base model, further trained with 60M tweets, randomly collected, resulting in a Twitter-domain adapted version. We use a selection of four TWEETEVAL models, each fine tuned for a twitter-specific downstream task: hate speech-, emotion-and irony-detection, and offensive language identification.…”