SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

Abstract

Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowledge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta.

Government-Led or Public-Led? Chinese Policy Agenda Setting during the COVID-19 Pandemic

“…Social Emotion Identification and Distribution.. We adopted the SKEP model (Tian et al 2020) for sentiment analysis. SKEP is a deep learning model, pre-trained with a corpus that contains over 3.2 million documents and yields state-of-the-art performance on several sentiment analysis benchmarks (especially the Chinese corpus).…”

Using social media data to assess the impact of COVID-19 on mental health in China

“…Using these data, we conducted our sentiment analysis by applying the SKEP model (Tian et al, 2020a) from Baidu Senta (an open-source python library) published in 2020, which integrated sentiment knowledge into pre-trained models and achieved new state-of-the-art results on most of the test data sets. For each tweet, the sentiment analysis could return two probabilities representing the intensity of the positive and negative emotions based on the text, and the sum of these two probabilities is 1.…”

“…The data included around 13 million geotagged tweets in mainland China between 1 January 2020 and 1 March 2020 from active Weibo users. For each tweet, we conducted the sentiment analysis to extract the expressed sentiment using the open-source NLP technique from Baidu (Tian et al, 2020a). Then we measured the daily mental health status for a city by calculating the median sentiment value based on tweets in that city on each day (Zheng, Wang, Sun, Zhang, & Kahn, 2019), which ranges from 0 to 1 with 0 indicating a strongly negative emotion and 1 indicating a strongly positive emotion.…”

Modeling Considerations for Quantitative Social Science Research Using Social Media Data

“…These language models can perform relatively well with some task-specific fine-tuning. Some relevant prediction tasks that have made use of these language models with social media data sets include demographic inference (Liu et al, 2021), stance detection (Kawintiranon & Singh, 2021;Ghosh et al, 2019), sentiment (Tian et al, 2020), and bot identification (Kudugunta & Ferrara, 2018).…”