Injecting User Identity Into Pretrained Language Models for Document-Level Sentiment Classification

Abstract

This paper mainly studies the combination of pre-trained language models and user identity information for document-level sentiment classification. In recent years, pre-trained language models (PLMs) such as BERT have achieved state-of-the-art results on many NLP applications, including document-level sentiment classification. On the other hand, a collection of works introduce additional information such as user identity for better text modeling. However, most of them inject user identity into traditional models, while few studies have been conducted to study the combination of pre-trained language models and user identity for even better performance. To address this issue, in this paper, we propose to unite user identity and PLMs and formulate User-enhanced Pre-trained Language Models (U-PLMs). Specifically, we demonstrate two simple yet effective attempts, i.e. embedding-based and attention-based personalization, which inject user identity into different parts of a pre-trained language model and provide personalization from different perspectives. Experiments in three datasets with two backbone PLMs show that our proposed methods outperform the best state-of-the-art baseline method with an absolute improvement of up to 3%, 2.8%, and 2.2% on accuracy. In addition, our methods encode user identity with plugin modules, which are fully compatible with most auto-encoding pre-trained language models.

CNO-LSTM: A Chaotic Neural Oscillatory Long Short-Term Memory Model for Text Classification

“…Generally, it includes Sentiment Analysis (SA), News Classification (CA), Topic Classification (TC) and Question Answering (QA) to reasoning and Natural Language Inference (NLI) with binary classification, multi-classification and multi-label classification. [2] For example, 1) sentiment analysis has simple happy and sad emotion categories [3][4] [5][6] [7], 2) email spam filter is considered as binary classification problem to distinguish email contents based on texts understanding for further classification, 3) publication database with multi-label such as WOS-11967 illustrated by index terms is classification of different topics in research areas.…”