Prospective Evaluation of Adverse Event Recognition Systems in Twitter: Results from the Web-RADR Project

Abstract

Introduction A large number of studies on systems to detect and sometimes normalize adverse events (AEs) in social media have been published, but evidence of their practical utility is scarce. This raises the question of the transferability of such systems to new settings. Objectives The aims of this study were to develop an AE recognition system, prospectively evaluate its performance on an external benchmark dataset and identify potential factors influencing the transferability of AE recognition systems. Methods A pipeline based on dictionary lookups and logistic regression classifiers was developed using a proprietary dataset of 196,533 Tweets manually annotated for AE relations and prospectively evaluated the system on the publicly available WEB-RADR reference dataset, exploring different aspects affecting transferability. Results Our system achieved 0.53 precision, 0.52 recall and 0.52 F1-score on the development test set; however, when applied to the WEB-RADR reference dataset, system performance dropped to 0.38 precision, 0.20 recall and 0.26 F1-score. Similarly, a previously published method aiming at automatically detecting adverse event posts reported 0.5 precision, 0.92 recall and 0.65 F1-score on thus another dataset, while performance on the WEB-RADR reference dataset was reduced to 0.37 precision, 0.63 recall and 0.46 F1-score. We identified four potential factors leading to poor transferability: overfitting, selection bias, label bias and prevalence. Conclusion We warn the community about a potentially large discrepancy between the expected performance of automated AE recognition systems based on published results and the actual observed performance on independent data. This study highlights the difficulty of implementing an all-purpose system for automatic adverse event recognition in Twitter, which could explain the lack of such systems in practical pharmacovigilance settings. Our recommendation is to use benchmark independent datasets, such as the WEB-RADR reference, to investigate the transferability of the adverse event recognition systems and ultimately enforce rigorous comparisons across studies on the task.

Mild Adverse Events of Sputnik V Vaccine Extracted from Russian Language Telegram Posts via BERT Deep Learning Model

“…To date, most of these studies processed data collected from Twitter [13][14][15][16][17][18][19][20][21][22]. Although social media platforms such as Twitter and Facebook are used in Russia, the most popular platforms are VK.com, which has about 97 million active monthly users [23], and Telegram Messenger, which is ranked second in the Russian Appstore, having 500 million active users overall.…”

“…Accounts of adverse reactions to drugs have been widely extracted from social media [11] in the context of mining consumer reviews on the Internet [12]. To date, most of these studies processed data collected from Twitter [13–22]. Although social media platforms such as Twitter and Facebook are used in Russia, the most popular platforms are VK.com, which has about 97 million active monthly users [23], and Telegram Messenger, which is ranked second in the Russian Appstore, having 500 million active users overall.…”

Pharmacovigilance and the digital world in Italy: presentation of the results of a national survey

“…7 The potential for analysing and managing this incessant flow of information lies in the development of artificial intelligence (AI) systems, which will help not only the identification of reports but also the control of their quality and their follow-up. 8 – 10 Projects in this direction are being set up 11 , 12 , 19 , 20 and are hampered by costs, by technical difficulties 21 and probably by the lack of clear regulatory guidance for the digital and social media area. Pharmacovigilance legislation, and regulations concerning other healthcare products, such as medical device, appears to be lagging behind the rapid development of the sector.…”

Incorporating Unstructured Patient Narratives and Health Insurance Claims Data in Pharmacovigilance: Natural Language Processing Analysis of Patient-Generated Texts About Systemic Lupus Erythematosus

“…To expand the scope of pharmacovigilance to patients' viewpoints, it is necessary to include data sources that can be used to analyze patient situations. Several studies [4][5][6] have explored the use of web-based resources such as Twitter in pharmacovigilance to include patients' viewpoints. Similarly, in Japan, we previously examined Japanese-language disease blogs (tōbyōki) as a resource for patient-generated data from the internet to augment pharmacovigilance [7].…”

“…Although several studies [ 4 , 8 ] have reported the utility of patient-derived data from the web for pharmacovigilance, concern over the effect of irrelevant data (ie, noise) has led some researchers to recommend that these data alone should not be used to derive pharmacovigilance statistics [ 9 ]. Using data from additional sources is one way to minimize this effect.…”