They May Not Work! An evaluation of eleven sentiment analysis tools on seven social media datasets

He, Lu; Yin, Tingjue; Zheng, Kai

doi:10.1016/j.jbi.2022.104142

Cited by 20 publications

(17 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Off-the-shelf sentiment analysis models include Amazon Web Services Comprehend sentiment analysis [ 53 ] and VADER [ 54 - 60 ], which is a Python lexicon and rule-based sentiment analysis tool [ 43 ]. In a recent evaluation of 11 sentiment analysis tools on 7 social media data sets, He et al [ 61 ] observed that these tools do not provide results that are accurate enough to aid in public health decision-making. Among all tools, VADER was one of the 3 top performers.…”

Section: Discussionmentioning

confidence: 99%

Users’ Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts

Dupuy-Zini¹,

Gérardin²,

Duclos³

et al. 2023

J Med Internet Res

View full text Add to dashboard Cite

Background Within a few months, the COVID-19 pandemic had spread to many countries and had been a real challenge for health systems all around the world. This unprecedented crisis has led to a surge of online discussions about potential cures for the disease. Among them, vaccines have been at the heart of the debates and have faced lack of confidence before marketing in France. Objective This study aims to identify and investigate the opinions of French Twitter users on the announced vaccines against COVID-19 through sentiment analysis. Methods This study was conducted in 2 phases. First, we filtered a collection of tweets related to COVID-19 available on Twitter from February 2020 to August 2020 with a set of keywords associated with vaccine mistrust using word embeddings. Second, we performed sentiment analysis using deep learning to identify the characteristics of vaccine mistrust. The model was trained on a hand-labeled subset of 4548 tweets. Results A set of 69 relevant keywords were identified as the semantic concept of the word “vaccin” (vaccine in French) and focused mainly on conspiracies, pharmaceutical companies, and alternative treatments. Those keywords enabled us to extract nearly 350,000 tweets in French. The sentiment analysis model achieved 0.75 accuracy. The model then predicted 16% of positive tweets, 41% of negative tweets, and 43% of neutral tweets. This allowed us to explore the semantic concepts of positive and negative tweets and to plot the trends of each sentiment. The main negative rhetoric identified from users’ tweets was that vaccines are perceived as having a political purpose and that COVID-19 is a commercial argument for the pharmaceutical companies. Conclusions Twitter might be a useful tool to investigate the arguments for vaccine mistrust because it unveils political criticism contrasting with the usual concerns on adverse drug reactions. As the opposition rhetoric is more consistent and more widely spread than the positive rhetoric, we believe that this research provides effective tools to help health authorities better characterize the risk of vaccine mistrust.

show abstract

Section: Discussionmentioning

confidence: 99%

Users’ Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts

Dupuy-Zini¹,

Gérardin²,

Duclos³

et al. 2023

J Med Internet Res

View full text Add to dashboard Cite

show abstract

“…The datasets were selected based on our systematic review and must include human coders' annotation as ground truth labels. 5 More detailed description of the datasets can be found in He et al 4 The evaluation datasets included four health-related social media datasets: Health Care Reform ("HCR"), Human Papillomavirus ("HPV") Vaccine, COVID-19 Masking ("Mask"), and Vitals.com Physician Reviews ("Vitals"). A non-health dataset, the IMDB Dataset ("IMDB") created from movie reviews, was used for baseline comparison.…”

Section: Evaluation Datasetsmentioning

confidence: 99%

Using Large Language Models for sentiment analysis of health-related social media data: empirical evaluation and practical tips

He,

Omranian,

McRoy

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Health-related social media data generated by patients and the public provide valuable insights into patient experiences and opinions toward health issues such as vaccination and medical treatments. Using Natural Language Processing (NLP) methods to analyze such data, however, often requires high-quality annotations that are difficult to obtain. The recent emergence of Large Language Models (LLMs) such as the Generative Pre-trained Transformers (GPTs) has shown promising performance on a variety of NLP tasks in the health domain with little to no annotated data. However, their potential in analyzing health-related social media data remains underexplored. In this paper, we report empirical evaluations of LLMs (GPT-3.5-Turbo, FLAN-T5, and BERT-based models) on a common NLP task of health-related social media data: sentiment analysis for identifying opinions toward health issues. We explored how different prompting and fine-tuning strategies affect the performance of LLMs on social media datasets across diverse health topics, including Healthcare Reform, vaccination, mask wearing, and healthcare service quality. We found that LLMs outperformed VADER, a widely used off-the-shelf sentiment analysis tool, but are far from being able to produce accurate sentiment labels. However, their performance can be improved by data-specific prompts with information about the context, task, and targets. The highest performing LLMs are BERT-based models that were fine-tuned on aggregated data. We provided practical tips for researchers to use LLMs on health-related social media data for optimal outcomes. We also discuss future work needed to continue to improve the performance of LLMs for analyzing health-related social media data with minimal annotations.

show abstract

“…𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑠𝑢𝑚 𝑐 𝑖𝑛 𝐶 𝑇𝑟𝑢𝑒𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠_𝑐 /𝑠𝑢𝑚 𝑐 𝑖𝑛 𝐶 (𝑇𝑟𝑢𝑒𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠_𝑐 + 𝐹𝑎𝑙𝑠𝑒𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠_𝑐) (5) After calculation precision and recall, F-measure is also calculated. F-score is the widely accepted measure to verify the accuracy of imbalanced classifications [30]. The harmonic mean of the two fractions precision and recall give the F1-score.…”

Section: 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑠𝑢𝑚 𝑐 𝑖𝑛 𝐶 𝑇𝑟𝑢𝑒𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠_𝑐 /𝑠𝑢𝑚 𝑐 𝑖𝑛 𝐶 (𝑇𝑟𝑢𝑒𝑃𝑜𝑠𝑖...mentioning

confidence: 99%

Reduction of false negatives in multi-class sentiment analysis

Aloysius

Selvan

2023

Bulletin EEI

View full text Add to dashboard Cite

Sentiment analysis classifications are done as positive, negative, as well as neutral ones. The increased usage of social media and its effects on society call for a more thorough, fine-grained explanation than that. In this study, classification is done in five classes-strongly positive, weakly positive, neutral, weakly negative, and strongly negative-in a more precise manner. Instead of using the typical ways of measuring accuracy alone, a novel method to eliminate false negatives (FN) is focused together with a fine-grained categorization. A bigger risk in sentiment analysis is a false negative. FN classification occurs when the context's polarity is identified as True when it is actually false. A complex dataset is used in this research for the experimental study, and the entire dataset is separated into five classes. Each class's FN are assessed using the suggested methodology. Comparing the proposed strategy to other, it was found to achieve about 53% more reduction in FN cases than rule based models and better predictions than compared machine learning models.

show abstract

They May Not Work! An evaluation of eleven sentiment analysis tools on seven social media datasets

Cited by 20 publications

References 43 publications

Users’ Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts

Users’ Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts

Using Large Language Models for sentiment analysis of health-related social media data: empirical evaluation and practical tips

Reduction of false negatives in multi-class sentiment analysis

Contact Info

Product

Resources

About