Lancaster A at SemEval-2017 Task 5: Evaluation metrics matter: predicting sentiment from financial news headlines

Abstract

This paper describes our participation in Task 5 track 2 of SemEval 2017 to predict the sentiment of financial news headlines for a specific company on a continuous scale between -1 and 1. We tackled the problem using a number of approaches, utilising a Support Vector Regression (SVR) and a Bidirectional Long Short-Term Memory (BLSTM). We found an improvement of 4-6% using the LSTM model over the SVR and came fourth in the track. We report a number of different evaluations using a finance specific word embedding model and reflect on the effects of using different evaluation metrics.

SemEval-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News

“…Multiple techniques were used by some authors in order to find the best one to use in this competition within their system. Moore and Rayson (2017) experimented with an ML and DL algorithm respectively, with the latter performing better. On the other hand, Cabanski et al (2017) implemented two-hybrid techniques, where the Hybrid (DL, Lex) approach produced their best result for this track , same as for track 1.…”

“…The Deep Learning-based techniques made use of the following algorithms: • RNN : Bidirectional Long Short-Term Memory (BLSTM) -adopted by Moore and Rayson (2017) • Bidirectional Gated Recurrent Unit (Bi-GRU) -adopted by Kar et al (2017) The CNN algorithm was the most popular amongst all Deep Learning-based techniques, with both systems ranking first (Mansar et al, 2017) and second (Kar et al, 2017) using it.…”

“…Some authors constructed their own lexica from external sources, such as Moore and Rayson (2017) (rank four) who manually downloaded 189,206 financial articles which contain 161,877,425 tokens from Factiva 17 (articles come from sources such as Financial Times that relate to United States companies only).…”

“…Novel features specific to the task included the creation of a domain-specific ontology (Schouten et al, 2017), a stocktwits-based embedding model and distance supervision model (Li, 2017) and domain-specific lexica (Moore and Rayson, 2017). Moreover, due to the emphasis of the task on the sentiment classification on a continuous scale, many approaches targeted regression-based models.…”

Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)

“…The approaches proposed by the participating systems explored a combination of machine learning methods using lexical features, sentiment lexical resources (both generic and specific to finance) and pre-trained word embedding models. Novel features specific to the task included the creation of a domain-specific ontology (Schouten et al, 2017), a stocktwits-based embedding model and distance supervision model (Li, 2017) and domain-specific lexica (Moore and Rayson, 2017). Moreover, due to the emphasis of the task on the sentiment classification on a continuous scale, many approaches targeted regression-based models.…”