Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection

Wahle, Jan Philip; Ashok, Nischal; Ruas, Terry; Meuschke, Norman; Ghosal, Tirthankar; Gipp, Béla

doi:10.22541/au.167528154.41917807/v1

Cited by 3 publications

(2 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ultimately, such language representation models would be used in the real world, not only in multiple‐choice settings, but also in so‐called generative settings where the model may be expected to generate answers to questions (without being given options). Even in the multiple‐choice setting, without robust commonsense, the model will likely not be usable for actual decision making unless we can trust that it is capable of generalization (Kejriwal, 2021; Misra, 2022; Wahle et al, 2022). One option to implementing such robustness in practice may be to add a ‘decision‐making layer’ on a pre‐trained language representation model rather than aim to modify the model's architecture from scratch 13 (Hong et al, 2021; Tang & Kejriwal, 2022; Zaib et al, 2020).…”

Section: Discussionmentioning

confidence: 99%

An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

Shen

Kejriwal

2023

Expert Systems

View full text Add to dashboard Cite

In the last 5 years, language representation models, such as BERT and GPT-3, based on transformer neural networks, have led to enormous progress in natural language processing (NLP). One such NLP task is commonsense reasoning, where performance is usually evaluated through multiple-choice question answering benchmarks. Till date, many such benchmarks have been proposed, and 'leaderboards' tracking stateof-the-art performance on those benchmarks suggest that transformer-based models are approaching human-like performance. Because these are commonsense benchmarks, however, such a model should be expected to generalize, that is, at least in aggregate, should not exhibit excessive performance loss across independent commonsense benchmarks regardless of the specific benchmark on (the training set of) which it has been fine-tuned. In this article, we evaluate this expectation by proposing a methodology and experimental study to measure the generalization ability of language representation models using a rigorous and intuitive metric. Using five established commonsense reasoning benchmarks, our experimental study shows that the models do not generalize well, and may be (potentially) susceptible to issues such as dataset bias. The results therefore suggest that current performance on benchmarks may be an over-estimate, especially if we want to use such models on novel commonsense problems for which a 'training' dataset may not be available, for the language representation model, to fine-tune on.

show abstract

Section: Discussionmentioning

confidence: 99%

An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

Shen

Kejriwal

2023

Expert Systems

View full text Add to dashboard Cite

show abstract

“…Recently, the NLP community adapted and extended the neural language model BERT [8] for a variety of tasks [2,5,33,34,45,49,54], similar to the way that word2vec [31] has influenced many later models in NLP [4,41,42]. Based on the Transformer architecture [48], BERT employs two pre-training tasks, i.e., Masked Language Model (MLM) and Next Sentence Prediction (NSP), to capture general aspects of language.…”

Section: Related Workmentioning

confidence: 99%

Identifying Machine-Paraphrased Plagiarism

Wahle

Ruas²,

Foltýnek³

et al. 2023

Preprint

View full text Add to dashboard Cite

Employing paraphrasing tools to conceal plagiarized text is a severe threat to academic integrity. To enable the detection of machine-paraphrased text, we evaluate the effectiveness of five pre-trained word embedding models combined with machine learning classifiers and state-of-the-art neural language models. We analyze preprints of research papers , graduation theses, and Wikipedia articles, which we paraphrased using different configurations of the tools SpinBot and SpinnerChief. The best performing technique, Longformer, achieved an average F1 score of 80.99% (F1=99.68% for SpinBot and F1=71.64% for Spinner-Chief cases), while human evaluators achieved F1=78.4% for SpinBot and F1=65.6% for SpinnerChief cases. We show that the automated classification alleviates shortcomings of widely-used text-matching systems , such as Turnitin and PlagScan. To facilitate future research, all data 3 , code 4 , and two web applications 56 showcasing our contributions are openly available.

show abstract

Do We Need a Specific Corpus and Multiple High-Performance GPUs for Training the BERT Model? An Experiment on COVID-19 Dataset

Nuntachit

Sugunnasil

2022

MAKE

View full text Add to dashboard Cite

The COVID-19 pandemic has impacted daily lives around the globe. Since 2019, the amount of literature focusing on COVID-19 has risen exponentially. However, it is almost impossible for humans to read all of the studies and classify them. This article proposes a method of making an unsupervised model called a zero-shot classification model, based on the pre-trained BERT model. We used the CORD-19 dataset in conjunction with the LitCovid database to construct new vocabulary and prepare the test dataset. For NLI downstream task, we used three corpora: SNLI, MultiNLI, and MedNLI. We significantly reduced the training time by 98.2639% to build a task-specific machine learning model, using only one Nvidia Tesla V100. The final model can run faster and use fewer resources than its comparators. It has an accuracy of 27.84%, which is lower than the best-achieved accuracy by 6.73%, but it is comparable. Finally, we identified that the tokenizer and vocabulary more specific to COVID-19 could not outperform the generalized ones. Additionally, it was found that BART architecture affects the classification results.

show abstract

Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection

Cited by 3 publications

References 26 publications

An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

Identifying Machine-Paraphrased Plagiarism

Do We Need a Specific Corpus and Multiple High-Performance GPUs for Training the BERT Model? An Experiment on COVID-19 Dataset

Contact Info

Product

Resources

About