An Analysis of French-Language Tweets About COVID-19 Vaccines: Supervised Learning Approach

Abstract

Background As the COVID-19 pandemic progressed, disinformation, fake news, and conspiracy theories spread through many parts of society. However, the disinformation spreading through social media is, according to the literature, one of the causes of increased COVID-19 vaccine hesitancy. In this context, the analysis of social media posts is particularly important, but the large amount of data exchanged on social media platforms requires specific methods. This is why machine learning and natural language processing models are increasingly applied to social media data. Objective The aim of this study is to examine the capability of the CamemBERT French-language model to faithfully predict the elaborated categories, with the knowledge that tweets about vaccination are often ambiguous, sarcastic, or irrelevant to the studied topic. Methods A total of 901,908 unique French-language tweets related to vaccination published between July 12, 2021, and August 11, 2021, were extracted using Twitter’s application programming interface (version 2; Twitter Inc). Approximately 2000 randomly selected tweets were labeled with 2 types of categorizations: (1) arguments for (pros) or against (cons) vaccination (health measures included) and (2) type of content (scientific, political, social, or vaccination status). The CamemBERT model was fine-tuned and tested for the classification of French-language tweets. The model’s performance was assessed by computing the F1-score, and confusion matrices were obtained. Results The accuracy of the applied machine learning reached up to 70.6% for the first classification (pro and con tweets) and up to 90% for the second classification (scientific and political tweets). Furthermore, a tweet was 1.86 times more likely to be incorrectly classified by the model if it contained fewer than 170 characters (odds ratio 1.86; 95% CI 1.20-2.86). Conclusions The accuracy of the model is affected by the classification chosen and the topic of the message examined. When the vaccine debate is jostled by contested political decisions, tweet content becomes so heterogeneous that the accuracy of the model drops for less differentiated classes. However, our tests showed that it is possible to improve the accuracy by selecting tweets using a new method based on tweet length.

Users’ Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts

“…In previous work, several custom algorithms were proposed, such as (1) deep learning with a CamemBERT model [ 28 ], BERT [ 46 , 47 ], RoBERTa [ 48 ], FastText [ 49 ], convolutional neural network–long short-term memory with word2vec embeddings [ 50 ] or (2) machine learning with naïve Bayes [ 51 ] or decision tree [ 52 ] models. Off-the-shelf sentiment analysis models include Amazon Web Services Comprehend sentiment analysis [ 53 ] and VADER [ 54 - 60 ], which is a Python lexicon and rule-based sentiment analysis tool [ 43 ].…”

“…Mønsted and Lehmann [ 49 ] obtained a micro-averaged F 1 -score of 0.762 for a 3-class prediction, which is a good result considering that the authors used Amazon’s Mechanical Turk platform. Sauvayre et al [ 28 ] classified tweets according to the opinion for or against users and obtained an accuracy of 0.706 using a fine-tuned CamemBERT prediction model with 2 classes, whereas we used 3 classes, which is more unfavorable. Kummervold et al [ 46 ] obtained an F 1 -score of 0.78 to predict the attitude of pregnant women toward vaccination against COVID-19, but the categories for the classification were different.…”

“…The objective was to perform unsupervised learning, while we focused on supervised learning. Blanc et al [ 27 ] compared the performances of CamemBERT and FlauBERT to build a chatbot, and Sauvayre et al [ 28 ] used CamemBERT to classify the opinion of Twitter users on COVID-19 vaccines. The BERT language model is therefore usable for the French language and could be adapted for the classification of tweets.…”

Identification, analysis and prediction of valid and false information related to vaccines from Romanian tweets

“…The obtained information, after manually analyzed, if presented through a web platform, could further aid in raising awareness regarding valid information, fake news content, as well as irrelevant information related to vaccines and shared through Twitter platform ( 20 , 34 ). With regards to the machine learning validation results obtained in other studies, a relevant comparison with the ones from the current studies would be difficult, since the majority of the studies which used vaccine related Twitter content reported the F1 Score as the most important classification evaluation metric ( 12 – 17 , 37 , 47 , 48 ). The F1 Score was computed in the current study as well and can be regarded as an acceptable balanced measure between precision and recall.…”

“…That is the main reason for which the focus in the current study, when evaluating the classification performance of the 6 algorithms, was put on the Matthews Correlation Coefficient ( 36 , 37 ). Moreover, the reported F1 Scores showed a high degree of variability, some studies reported F1 Scores of under 0.6, while others reported enhanced values, of 0.7–0.8 and others obtained almost perfect values, of over 0.95, while the implemented machine learning algorithms included both classical (such as Random Forest and SVM) and newer model types (such as deep learning and BERT) applied on various languages, such as English, French, Dutch or Moroccan ( 12 – 17 , 47 , 48 ). As a comparison, the maximum F1 Score which was obtained in the current study SVM ranged from 0.542 (internal period validation – SVM + MLP Ensemble) to 0.658 (cross-validation – BERT) and 0.655 (external validation – SVM).…”