A multi-source entity-level sentiment corpus for the financial domain: the FinLin corpus

Daudert, Tobias

doi:10.1007/s10579-021-09555-3

Cited by 4 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• FinLin [3]: Investor reports, company reports, news articles, and microblogs from multiple sources annotated using a two-dimensional scale of sentiment score and relevance score by three annotators with specific profiles. The annotations are consolidated either automatically or manually by an additional annotator using the same tool (AWOCATo).…”

Section: Related Workmentioning

confidence: 99%

“…The study found evidence for the supremacy of the reader's perspective in terms of Inter-Annotator Agreement (IAA) and rating intensity, and achieved near-human performance when mapping between dimensional and categorical formats. Other works are [8], [9], [10], [11], [12], [13], [14], [15]. Some of the benefits of these methodologies are: They cover a variety of domains and topics, such as movie reviews, tweets, financial texts, news articles, and video data; they use different annotation schemes and tools to capture different aspects of sentiment, such as polarity, intensity, relevance, event type, sector, asset type, and transaction anonymity.…”

Section: Related Workmentioning

confidence: 99%

“…As observed in [3], sentiment annotation requires a relevance score as well as the sentiment score itself if data is collected from generic sources for credibility. To avoid this step, we followed the pipeline set by [4] which used a leading news aggregator (in their case: Yahoo!…”

Section: Corpus Designmentioning

confidence: 99%

“…The column 'reasoning' provides an explanation on the combined decision. The color green represents an easy decision, while the colors yellow and red represent decisions that have some sort of disagreement between annotators, but instead of using a 4th annotator in controversial cases as done by [3], we decided to set the annotation equal to 0 (neutral) as the number of cases is small and, as we will see later, both inter-annotator agreement study and pre-trained sentiment analysis show effective quality of the annotation as it is.…”

Section: Annotation Definitions and Proceduresmentioning

confidence: 99%

See 3 more Smart Citations

Annotators’ Selection Impact on the Creation of a Sentiment Corpus for the Cryptocurrency Financial Domain

Gadi,

Sicilia

2023

IEEE Access

View full text Add to dashboard Cite

Well labeled natural language corpus data is essential for most natural language processing techniques, especially in specialized fields. However, cohort biases remain a significant challenge in machine learning. The narrow origin of data sampling or human annotators in cohorts is a prevalent issue for machine learning researchers due to its potential to induce bias in the final product. During the development of the CryptoLin corpus for another research project, the authors became concerned about the potential influence of cohort bias on the selection of annotators. Therefore, this paper addresses the question of whether cohort diversity improves the labeling result through the implementation of a repeated annotator process, involving two annotator cohorts and a statistically robust comparison methodology. The utilization of statistical tests, such as the Chi-Square Independence test for absolute frequency tables, and the construction of confidence intervals for Kappa point estimates, facilitates a rigorous analysis of the differences between Kappa estimates. Furthermore, the application of a two-proportion z-test to compare the accuracy scores of UTAD and IE annotators for various pre-trained models, including Vader Sentiment Analysis, TextBlob Sentiment Analysis, Flair NLP library, and FinBERT Financial Sentiment Analysis with BERT, contributes to the advancement of knowledge in this field. The paper utilizes Cryptocurrency Linguo (CryptoLin), a corpus containing 2683 cryptocurrency-related news articles spanning more than three years,and compares two different selection criteria for the annotators. CryptoLin was annotated twice with discrete values representing negative, neutral, and positive news respectively. The first annotation was done by twenty-seven annotators from the same cohort. Each news title was randomly assigned and blindly annotated by three human annotators. The second annotation was carried out by eighty-three annotators from three cohorts. Each news title was randomly assigned and blindly annotated by three human annotators, one in each different cohort. In both annotations, a consensus mechanism using simple voting was applied. The first annotation used the same cohort with students from the same nationality and background. The second used three cohorts with students from a very diverse set of nationalities and educational backgrounds. The results demonstrate that manual labeling done by both groups was acceptable according to inter-rater reliability coefficients Fleiss's Kappa, Krippendorff's Alpha, and Gwet's AC1. Preliminary analysis utilizing Vader, Textblob, Flair, and FinBERT confirmed the utility of the data set labeling for further refinement of sentiment analysis algorithms. Our results also highlight that the more diverse annotator pool performed better in all measured aspects.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Corpus Designmentioning

confidence: 99%

Section: Annotation Definitions and Proceduresmentioning

confidence: 99%

See 2 more Smart Citations

Annotators’ Selection Impact on the Creation of a Sentiment Corpus for the Cryptocurrency Financial Domain

Gadi,

Sicilia

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

A sentiment corpus for the cryptocurrency financial domain: the CryptoLin corpus

Gadi,

Sicilia

2024

Lang Resources & Evaluation

View full text Add to dashboard Cite

The objective of this paper is to describe Cryptocurrency Linguo (CryptoLin), a novel corpus containing 2683 cryptocurrency-related news articles covering more than a three-year period. CryptoLin was human-annotated with discrete values representing negative, neutral, and positive news respectively. Eighty-three people participated in the annotation process; each news title was randomly assigned and blindly annotated by three human annotators, one in each different cohort, followed by a consensus mechanism using simple voting. The selection of the annotators was intentionally made using three cohorts with students from a very diverse set of nationalities and educational backgrounds to minimize bias as much as possible. In case one of the annotators was in total disagreement with the other two (e.g., one negative vs two positive or one positive vs two negative), we considered this minority report and defaulted the labeling to neutral. Fleiss’s Kappa, Krippendorff’s Alpha, and Gwet’s AC1 inter-rater reliability coefficients demonstrate CryptoLin’s acceptable quality of inter-annotator agreement. The dataset also includes a text span with the three manual label annotations for further auditing of the annotation mechanism. To further assess the quality of the labeling and the usefulness of CryptoLin dataset, it incorporates four pretrained Sentiment Analysis models: Vader, Textblob, Flair, and FinBERT. Vader and FinBERT demonstrate reasonable performance in the CryptoLin dataset, indicating that the data was not annotated randomly and is therefore useful for further research1. FinBERT (negative) presents the best performance, indicating an advantage of being trained with financial news. Both the CryptoLin dataset and the Jupyter Notebook with the analysis, for reproducibility, are available at the project’s Github. Overall, CryptoLin aims to complement the current knowledge by providing a novel and publicly available Gadi and Ángel Sicilia (Cryptolin dataset and python jupyter notebooks reproducibility codes, 2022) cryptocurrency sentiment corpus and fostering research on the topic of cryptocurrency sentiment analysis and potential applications in behavioral science. This can be useful for businesses and policymakers who want to understand how cryptocurrencies are being used and how they might be regulated. Finally, the rules for selecting and assigning annotators make CryptoLin unique and interesting for new research in annotator selection, assignment, and biases.

show abstract

Exploiting textual and relationship information for fine-grained financial sentiment analysis

Daudert

2021

Knowledge-Based Systems

View full text Add to dashboard Cite

A multi-source entity-level sentiment corpus for the financial domain: the FinLin corpus

Cited by 4 publications

References 34 publications

Annotators’ Selection Impact on the Creation of a Sentiment Corpus for the Cryptocurrency Financial Domain

Annotators’ Selection Impact on the Creation of a Sentiment Corpus for the Cryptocurrency Financial Domain

A sentiment corpus for the cryptocurrency financial domain: the CryptoLin corpus

Exploiting textual and relationship information for fine-grained financial sentiment analysis

Contact Info

Product

Resources

About