Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection

Wahle, Jan Philip; Ashok, Nischal; Ruas, Terry; Meuschke, Norman; Ghosal, Tirthankar; Gipp, Béla

doi:10.1007/978-3-030-96957-8_33

Cited by 17 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The study from 2021 [53] evaluated fifteen transformer-based models to detect misinformation in several COVID-19 datasets, which are COVID-CQ [54], CoAID [55], Re-COVery [56], CMU-MisCov19 [57], and COVID19FN [58]. In this study, tokenizers and models tailored to COVID-19 data did not outperform general-purpose tokenizers and models, according to the researchers.…”

Section: Discussionmentioning

confidence: 94%

Do We Need a Specific Corpus and Multiple High-Performance GPUs for Training the BERT Model? An Experiment on COVID-19 Dataset

Nuntachit

Sugunnasil

2022

MAKE

View full text Add to dashboard Cite

The COVID-19 pandemic has impacted daily lives around the globe. Since 2019, the amount of literature focusing on COVID-19 has risen exponentially. However, it is almost impossible for humans to read all of the studies and classify them. This article proposes a method of making an unsupervised model called a zero-shot classification model, based on the pre-trained BERT model. We used the CORD-19 dataset in conjunction with the LitCovid database to construct new vocabulary and prepare the test dataset. For NLI downstream task, we used three corpora: SNLI, MultiNLI, and MedNLI. We significantly reduced the training time by 98.2639% to build a task-specific machine learning model, using only one Nvidia Tesla V100. The final model can run faster and use fewer resources than its comparators. It has an accuracy of 27.84%, which is lower than the best-achieved accuracy by 6.73%, but it is comparable. Finally, we identified that the tokenizer and vocabulary more specific to COVID-19 could not outperform the generalized ones. Additionally, it was found that BART architecture affects the classification results.

show abstract

Section: Discussionmentioning

confidence: 94%

Do We Need a Specific Corpus and Multiple High-Performance GPUs for Training the BERT Model? An Experiment on COVID-19 Dataset

Nuntachit

Sugunnasil

2022

MAKE

View full text Add to dashboard Cite

show abstract

“…Another study using the ReCOVery data set for model development explored the use of multiple languages for fake news detection to improve model performance [ 67 ]. Finally, Wahle et al [ 68 ] used the ReCOVery data set as 1 of 6 COVID-19 misinformation data sets to evaluate the performance of 15 transformer-based ML models to determine the generalizability of different transformer models. Differing from the aforementioned studies, we were able to demonstrate that the use of readability, text characteristics, sentiment, and lexical categories can improve upon the original ReCOVery data set baseline models [ 17 ].…”

Section: Discussionmentioning

confidence: 99%

Data Exploration and Classification of News Article Reliability: Deep Learning Study

Zhan¹,

Li²,

Osmani³

et al. 2022

JMIR Infodemiology

View full text Add to dashboard Cite

Background During the ongoing COVID-19 pandemic, we are being exposed to large amounts of information each day. This “infodemic” is defined by the World Health Organization as the mass spread of misleading or false information during a pandemic. This spread of misinformation during the infodemic ultimately leads to misunderstandings of public health orders or direct opposition against public policies. Although there have been efforts to combat misinformation spread, current manual fact-checking methods are insufficient to combat the infodemic. Objective We propose the use of natural language processing (NLP) and machine learning (ML) techniques to build a model that can be used to identify unreliable news articles online. Methods First, we preprocessed the ReCOVery data set to obtain 2029 English news articles tagged with COVID-19 keywords from January to May 2020, which are labeled as reliable or unreliable. Data exploration was conducted to determine major differences between reliable and unreliable articles. We built an ensemble deep learning model using the body text, as well as features, such as sentiment, Empath-derived lexical categories, and readability, to classify the reliability. Results We found that reliable news articles have a higher proportion of neutral sentiment, while unreliable articles have a higher proportion of negative sentiment. Additionally, our analysis demonstrated that reliable articles are easier to read than unreliable articles, in addition to having different lexical categories and keywords. Our new model was evaluated to achieve the following performance metrics: 0.906 area under the curve (AUC), 0.835 specificity, and 0.945 sensitivity. These values are above the baseline performance of the original ReCOVery model. Conclusions This paper identified novel differences between reliable and unreliable news articles; moreover, the model was trained using state-of-the-art deep learning techniques. We aim to be able to use our findings to help researchers and the public audience more easily identify false information and unreliable media in their everyday lives.

show abstract

“…BART is a two-stage denoising autoencoder that corrupts its input text and reconstructs it through a sequence-to-sequence model. We chose BART because of its ability to perform a wide range of downstream tasks, such as paraphrase detection (Wahle et al, 2022b), fake news identification (Wahle et al, 2022a), and text summarization (Lewis et al, 2020). Additionally, in our preliminary experiments, BART also performed better than other candidate models such as PEGASUS (Zhang et al, 2020) and T5 (Raffel et al, 2020) (comparison in Tables 9 and 10 in appendix B).…”

Section: Methodsmentioning

confidence: 99%

Analyzing Multi-Task Learning for Abstractive Text Summarization

Kirstein¹,

Wahle²,

Ruas³

et al. 2022

Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)

View full text Add to dashboard Cite

Despite the recent success of multi-task learning and pre-finetuning for natural language understanding, few works have studied the effects of task families on abstractive text summarization. Task families are a form of task grouping during the pre-finetuning stage to learn common skills, such as reading comprehension. To close this gap, we analyze the influence of multi-task learning strategies using task families for the English abstractive text summarization task. We group tasks into one of three strategies, i.e., sequential, simultaneous, and continual multi-task learning, and evaluate trained models through two downstream tasks. We find that certain combinations of task families (e.g., advanced reading comprehension and natural language inference) positively impact downstream performance. Further, we find that choice and combinations of task families influence downstream performance more than the training scheme, supporting the use of task families for abstractive text summarization. Our code is publicly available 1 .

show abstract

Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection

Cited by 17 publications

References 22 publications

Do We Need a Specific Corpus and Multiple High-Performance GPUs for Training the BERT Model? An Experiment on COVID-19 Dataset

Do We Need a Specific Corpus and Multiple High-Performance GPUs for Training the BERT Model? An Experiment on COVID-19 Dataset

Data Exploration and Classification of News Article Reliability: Deep Learning Study

Analyzing Multi-Task Learning for Abstractive Text Summarization

Contact Info

Product

Resources

About