RuSentiTweet: a sentiment analysis dataset of general domain tweets in Russian

Abstract

The Russian language is still not as well-resourced as English, especially in the field of sentiment analysis of Twitter content. Though several sentiment analysis datasets of tweets in Russia exist, they all are either automatically annotated or manually annotated by one annotator. Thus, there is no inter-annotator agreement, or annotation may be focused on a specific domain. In this article, we present RuSentiTweet, a new sentiment analysis dataset of general domain tweets in Russian. RuSentiTweet is currently the largest in its class for Russian, with 13,392 tweets manually annotated with moderate inter-rater agreement into five classes: Positive, Neutral, Negative, Speech Act, and Skip. As a source of data, we used Twitter Stream Grab, a historical collection of tweets obtained from the general Twitter API stream, which provides a 1% sample of the public tweets. Additionally, we released a RuBERT-based sentiment classification model that achieved F1 = 0.6594 on the test subset.

The voice of Twitter: observable subjective well-being inferred from tweets in Russian

“…Although several training datasets are available for sentiment analysis of texts from social networks for the Russian language ( Kotelnikov, 2021 ; Smetanin, 2020 ), only two consist of common-domain texts, are manually annotated, and report inter-annotator agreement scores. The first, RuSentiment ( Rogers et al, 2018 ), consists of general-domain texts from VKontake, and the second, RuSentiTweet ( Smetanin, 2022b ), consists of general-domain texts from Twitter.…”

“…At present, pre-trained language models have achieved the highest classification results on most Russian-language sentiment analysis datasets available publicly. For example, fine-tuned RuBERT achieved state-of-the-art results on RuSentiTweet ( Smetanin, 2022b ), LINIS Crowd ( Koltsova, Alexeeva & Kolcov, 2016 ), RuTweetCorp ( Rubtsova, 2013 ), and RuReviews ( Smetanin, 2022b ) datasets; fine-tuned RuRoBERTa-Large achieved SOTA results on RuSentiment ( Rogers et al, 2018 ). Thus, we fine-tuned a pre-trained language model for sentiment classification in this study.…”

“…Several articles ( Chizhic, 2016 ; Smetanin, 2017 ) have been published that consider Russian-language Twitter as a data source for the OSWB, but they were more concerned with the methodology or system for building the index; they omitted description of the resulting OSWB index itself. In the context of measuring OSWB based on the Russian-language Twitter segment, the absence of a significant number of existing studies is quite expected, as the first dataset of general-domain tweets in Russian with manual annotation ( Smetanin, 2022b ) appeared only in 2022. Thus, studies have not previously published an OSWB index based on the Russian-speaking segment of Twitter and have not compared it with an SWB index based on surveys.…”

“…Considering that the share of geotagged tweets in Russian is extremely low in Twitter Stream Grab (see “Geotagged Tweets in Russia”), we decided to focus on Russian-language tweets in general because they seem to predominantly consist of tweets published from Russia (see “Tweets in Russian and tweets from Russia”). Data were filtered as previously described in Smetanin (2022b) . Specifically, since Twitter Stream Grab consists of tweets in different languages, our next step was to remove tweets written in non-Russian languages.…”

“…As the training dataset for the classification model, we chose RuSentiTweet ( Smetanin, 2022b ), the largest general-domain tweet sentiment analysis dataset in Russian. RuSentiTweet was generated from Twitter Stream Grab—the same data source that we chose to use in our study—so this dataset is ideal for our study.…”