Removing Non-Stationary Knowledge From Pre-Trained Language Models for Entity-Level Sentiment Classification in Finance

Abstract

Extraction of sentiment signals for stock movement prediction from news text, stock message boards, and business reports have been a rising field of interest in finance. Building upon past literature, the most recent work attempt to better capture sentiment from sentences with complex syntactic structures by introducing aspect-level sentiment classification (ASC). Despite the growing interest, however, finegrained sentiment analysis has not been fully explored in non-English literature due to the shortage of annotated financespecific data. Accordingly, it is necessary for non-English languages to leverage datasets and pre-trained language models (PLM) of different domains, languages, and tasks to improve their performance. To facilitate finance-specific ASC research in the Korean language, we build KorF inASC, a Korean aspect-level sentiment classification dataset for finance consisting of 12,613 human-annotated samples, and explore methods of intermediate transfer learning. Our experiments indicate that past research has been ignorant towards the potentially wrong knowledge of financial entities encoded during the training phase, which has overestimated the predictive power of PLMs. In our work, we use the term "non-stationary knowledge" to refer to information that was previously correct but is likely to change, and present "TGT-Masking", a novel masking pattern to restrict PLMs from speculating knowledge of the kind. Finally, through a series of transfer learning with TGT-Masking applied we improve 22.63% of classification accuracy compared to standalone models on KorF inASC.