Developing a standardized protocol for computational sentiment analysis research using health-related social media data

Abstract

Objective Sentiment analysis is a popular tool for analyzing health-related social media content. However, existing studies exhibit numerous methodological issues and inconsistencies with respect to research design and results reporting, which could lead to biased data, imprecise or incorrect conclusions, or incomparable results across studies. This article reports a systematic analysis of the literature with respect to such issues. The objective was to develop a standardized protocol for improving the research validity and comparability of results in future relevant studies. Materials and Methods We developed the Protocol of Analysis of senTiment in Health (PATH) based on a systematic review that analyzed common research design choices and how such choices were made, or reported, among eligible studies published 2010-2019. Results Of 409 articles screened, 89 met the inclusion criteria. A total of 16 distinctive research design choices were identified, 9 of which have significant methodological or reporting inconsistencies among the articles reviewed, ranging from how relevance of study data was determined to how the sentiment analysis tool selected was validated. Based on this result, we developed the PATH protocol that encompasses all these distinctive design choices and highlights the ones for which careful consideration and detailed reporting are particularly warranted. Conclusions A substantial degree of methodological and reporting inconsistencies exist in the extant literature that applied sentiment analysis to analyzing health-related social media data. The PATH protocol developed through this research may contribute to mitigating such issues in future relevant studies.

Why do people oppose mask wearing? A comprehensive analysis of U.S. tweets during the COVID-19 pandemic

“…To classify public opinions toward mask wearing, we first applied the sentiment analysis approach which has been commonly used in the literature to study attitudes expressed in social media data. 30 We tested four off-the-shelf sentiment analysis tools that have been most commonly used: Valence Aware Dictionary for sEntiment Reasoning (VADER), 31 TextBlob, 32 Stanford Natural Language Processing (NLP), 33 and Linguistic Inquiry and Word Count (LIWC). 30,34,35 We manually annotated the sentiment of 500 random tweets and compared the results to the outputs of these tools.…”

“…30 We tested four off-the-shelf sentiment analysis tools that have been most commonly used: Valence Aware Dictionary for sEntiment Reasoning (VADER), 31 TextBlob, 32 Stanford Natural Language Processing (NLP), 33 and Linguistic Inquiry and Word Count (LIWC). 30,34,35 We manually annotated the sentiment of 500 random tweets and compared the results to the outputs of these tools. We found that none of these off-the-shelf tools were able to produce accurate attitude classifications, at least not in our study context.…”

Sentiment Analysis of Public Social Media as a Tool for Health-Related Topics

“…6, there has been a proportional increase in both interest and research efforts to process and interpret the sentiments and trends that are present in social media data. This substantial increase in research efforts has been identified to have a detrimental effect on the consistency of experimental designs, processing of data sets, and incomparability and/or misinterpretation of results [202]. To this end, the work of Hu et al [202] presents a standardized methodology for the structuring and reporting of future works in the field of SA, which could contribute significantly to the comparability of the results of individual works in this field.…”

“…The references studied in Section III were obtained by searching in different scientific search engines such as Scopus, Science Direct, IEEE Xplore, and Google Scholar, for the following keywords: Sentiment Analysis, Text Classification, Social Media, Social Networks, Health, Twitter, Facebook, Instagram, epidemic, pandemic, COVID-19, SARS-Cov-19, Influenza, Virus, Vaccine, Vaccination, Affliction, Mental Health, Mental Illness, Mental Disorder Psychiatric, Suicide, Depression, Public Sentiment, Discourse, and Public Health. These keywords were selected through a search of the most relevant literature on the subject of study of this work, following a literature search methodology based on the one presented in [202]. Upon obtaining a complete reference list, we proceeded to filter the literature based on a series of parameters.…”

ConsistSum

“…Social media generates a large amount of review data on the Internet that varies among topics, 𝑒.𝑔., online goods [34], government policies [25], and health care [16]. To facilitate the decisionmaking process, many opinion summarization techniques have been proposed to filter valuable opinions out of the massive review data [1,2,6,7,9,18,24,41,42,44].…”