MisRoBÆRTa: Transformers versus Misinformation

Abstract

Misinformation is considered a threat to our democratic values and principles. The spread of such content on social media polarizes society and undermines public discourse by distorting public perceptions and generating social unrest while lacking the rigor of traditional journalism. Transformers and transfer learning proved to be state-of-the-art methods for multiple well-known natural language processing tasks. In this paper, we propose MisRoBÆRTa, a novel transformer-based deep neural ensemble architecture for misinformation detection. MisRoBÆRTa takes advantage of two state-of-the art transformers, i.e., BART and RoBERTa, to improve the performance of discriminating between real news and different types of fake news. We also benchmarked and evaluated the performances of multiple transformers on the task of misinformation detection. For training and testing, we used a large real-world news articles dataset (i.e., 100,000 records) labeled with 10 classes, thus addressing two shortcomings in the current research: (1) increasing the size of the dataset from small to large, and (2) moving the focus of fake news detection from binary classification to multi-class classification. For this dataset, we manually verified the content of the news articles to ensure that they were correctly labeled. The experimental results show that the accuracy of transformers on the misinformation detection problem was significantly influenced by the method employed to learn the context, dataset size, and vocabulary dimension. We observe empirically that the best accuracy performance among the classification models that use only one transformer is obtained by BART, while DistilRoBERTa obtains the best accuracy in the least amount of time required for fine-tuning and training. However, the proposed MisRoBÆRTa outperforms the other transformer models in the task of misinformation detection. To arrive at this conclusion, we performed ample ablation and sensitivity testing with MisRoBÆRTa on two datasets.

It’s All in the Embedding! Fake News Detection Using Document Embeddings

“…Moreover, the transformer embeddings obtained the best results among the document embeddings experiments as they manage to encode and preserve the context within the vector representation. When compared to the state-of-the-art model MisRoBAERTa [23], the BiLSTM with BART obtained similar results, while the BiGRU with BART marginally outperformed the model with a 0.02% difference in accuracy. We hypothesize that this difference in performance is due to the use of pre-trained transformers instead of fine-tuned versions.…”

“…We employed Keras for implementing the neural models. For comparison, we used the free implementation of MisRoBAERTa [23], made available by the authors on GitHub.…”

“…Aslam et al [22] used an ensemble-based deep learning model that combines two architectures, i.e., Bi-LSTM-GRU-Dense and Dense. Truicȃ and Apostol [23] propose MisRoBAERTa, a BERT-and ROBERTA-based ensemble model for fake news detection.…”

“…For the fake news detection task, we employ the following algorithms to construct models: Naïve Bayes (NB), Gradient Boosted Trees (XGBTrees), Perceptron, Multi-Layer Perceptron (MLP), Long Short-Term Memory Network (LSTM), Bidirectional LSTM (BiLSTM), Gated Recurrent Units (GRU), and Bidirectional GRU (BiGRU). For comparison, we use MisRoBAERTa [23]. In the original article that presents MisRoBAERTa, the authors fine-tune both BART and ROBERTA.…”

“…For the experiments, we used a set of 20K English language news articles (10K reliable and 10K fake) selected from the Fake News Corpus [50] as it is widely used in current research [55,56,57,16,23]. Some of the labels might not be correct because the original dataset is not manually annotated.…”