Automatic Detection of Machine Generated Text: A Critical Survey

Jawahar, Ganesh; Abdul-Mageed, Muhammad; Lakshmanan, Laks V. S.

doi:10.18653/v1/2020.coling-main.208

Cited by 73 publications

(49 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Our work is also related to detecting machinegenerated and manipulated text (Jawahar et al, 2020;Nagoudi et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking

Alhindi¹,

Alabdulkarim²,

Alshehri³

et al. 2021

Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda

Self Cite

View full text Add to dashboard Cite

With the continuing spread of misinformation and disinformation online, it is of increasing importance to develop combating mechanisms at scale in the form of automated systems that support multiple languages. One task of interest is claim veracity prediction, which can be addressed using stance detection with respect to relevant documents retrieved online. To this end, we present our new Arabic Stance Detection dataset (AraStance) of 4,063 claimarticle pairs from a diverse set of sources comprising three fact-checking websites and one news website. AraStance covers false and true claims from multiple domains (e.g., politics, sports, health) and several Arab countries, and it is well-balanced between related and unrelated documents with respect to the claims. We benchmark AraStance, along with two other stance detection datasets, using a number of BERT-based models. Our best model achieves an accuracy of 85% and a macro F1 score of 78%, which leaves room for improvement and reflects the challenging nature of AraStance and the task of stance detection in general.

show abstract

“…Our work is also related to detecting machinegenerated and manipulated text (Jawahar et al, 2020;Nagoudi et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…Arabic tweets, which in turn use both Modern Standard Arabic and Dialectal Arabic (Abdul-Mageed et al, 2020).…”

Section: Marbert Which Is Trained On One Billionmentioning

confidence: 99%

AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking

Alhindi¹,

Alabdulkarim²,

Alshehri³

et al. 2021

Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda

Self Cite

View full text Add to dashboard Cite

show abstract

“…Automatic Detection of Generated Text Given the potential malicious applications of text generation (Solaiman et al, 2019), it is thus vital to build detectors to distinguish text generated by machines from humans (Gehrmann et al, 2019;Jawahar et al, 2020;Varshney et al, 2020;Çano and Bojar, 2020). Most current work focus on fake news detection (Rashkin et al, 2017;Zhou et al, 2019;Bhat and Parthasarathy, 2020;Zhong et al, 2020;Schuster et al, 2020;Ippolito et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation

Uchendu¹,

Ma²,

Le³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

Recent progress in generative language models has enabled machines to generate astonishingly realistic texts. While there are many legitimate applications of such models, there is also a rising need to distinguish machine-generated texts from human-written ones (e.g., fake news detection). However, to our best knowledge, there is currently no benchmark environment with datasets and tasks to systematically study the so-called "Turing Test" problem for neural text generation methods. In this work, we present the TURINGBENCH benchmark environment, which is comprised of (1) a dataset with 200K human-or machine-generated samples across 20 labels {Human,

show abstract

“…The baselines and models are trained on texts from the GPT-2 small model and further used to detect texts generated by unseen GPT-style models with pure sampling: GPT-2 Medium (345M), GPT-2 Large (762M) and GPT-2 XL (1542M). Note that such a setting is the most challenging as it requires the transfer from the smallest model to that of the higher number of parameters (Jawahar et al, 2020).…”

Section: Robustness Towards Unseen Modelsmentioning

confidence: 99%

Artificial Text Detection via Examining the Topology of Attention Maps

Kushnareva¹,

Cherniavskii²,

Mikhailov³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

The impressive capabilities of recent generative models to create texts that are challenging to distinguish from the human-written ones can be misused for generating fake news, product reviews, and even abusive content. Despite the prominent performance of existing methods for artificial text detection, they still lack interpretability and robustness towards unseen models. To this end, we propose three novel types of interpretable topological features for this task based on Topological Data Analysis (TDA) which is currently understudied in the field of NLP. We empirically show that the features derived from the BERT model outperform count-and neural-based baselines up to 10% on three common datasets, and tend to be the most robust towards unseen GPT-style generation models as opposed to existing methods. The probing analysis of the features reveals their sensitivity to the surface and syntactic properties. The results demonstrate that TDA is a promising line with respect to NLP tasks, specifically the ones that incorporate surface and structural information.

show abstract

Automatic Detection of Machine Generated Text: A Critical Survey

Cited by 73 publications

References 45 publications

AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking

AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking

TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation

Artificial Text Detection via Examining the Topology of Attention Maps

Contact Info

Product

Resources

About