A Watermark for Large Language Models

Kirchenbauer, John; Geiping, Jonas; Wen, Yuxin; Katz, Jonathan; Miers, Ian; Goldstein, Tom

doi:10.48550/arxiv.2301.10226

Cited by 44 publications

(89 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, tools, such as the recently released GPTZero [63], which uses perplexity, as a measure that hints at generalization capabilities (of the agent by which the text was written), to detect AI involvement in text writing, are expected to provide additional support. More advanced techniques aim at watermarking the content generated by language models [64,65], e.g., by biasing the content generation towards terms, which are rather unlikely to be jointly used by humans in a text passage. In the long run, however, we believe that developing curricula and instructions that encourage the creative and evidence-based use of large language models will be the key to solving this problem.…”

Section: • Incorporating Human Expertise and Teachers To Review Valid...mentioning

confidence: 99%

ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education

Kasneci¹,

Seßler²,

Küchemann³

et al. 2023

Preprint

133

View full text Add to dashboard Cite

Large language models represent a significant advancement in the field of AI. The underlying technology is key to further innovations and, despite critical views and even bans within communities and regions, large language models are here to stay. This position paper presents the potential benefits and challenges of educational applications of large language models, from student and teacher perspectives. We briefly discuss the current state of large language models and their applications. We then highlight how these models can be used to create educational content, improve student engagement and interaction, and personalize learning experiences. With regard to challenges, we argue that large language models in education require teachers and learners to develop sets of competencies and literacies necessary to both understand the technology as well as their limitations and unexpected brittleness of such systems. In addition, a clear strategy within educational systems and a clear pedagogical approach with a strong focus on critical thinking and strategies for fact checking are required to integrate and take full advantage of large language models in learning settings and teaching curricula. Other challenges such as the potential bias in the output, the need for continuous human oversight, and the potential for misuse are not unique to the application of AI in education. But we believe that, if handled sensibly, these challenges can offer insights and opportunities in education scenarios to acquaint students early on with potential societal biases, criticalities, and risks of AI applications. We conclude with recommendations for how to address these challenges and ensure that such models are used in a responsible and ethical manner in education.

show abstract

Section: • Incorporating Human Expertise and Teachers To Review Valid...mentioning

confidence: 99%

ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education

Kasneci¹,

Seßler²,

Küchemann³

et al. 2023

Preprint

133

View full text Add to dashboard Cite

show abstract

“…Since these approaches rely on a neural network for their detection, they can be vulnerable to adversarial and poisoning attacks [Goodfellow et al, 2014, Sadasivan et al, 2023, Kumar et al, 2022. Another line of work aims to watermark AI-generated texts to ease their detection [Atallah et al, 2001, Wilson et al, 2014, Kirchenbauer et al, 2023, Zhao et al, 2023. Watermarking eases the detection of LLM output text by imprinting specific patterns on them.…”

Section: Introductionmentioning

confidence: 99%

“…Watermarking eases the detection of LLM output text by imprinting specific patterns on them. Soft watermarking proposed in Kirchenbauer et al [2023] partitions tokens into green and red lists to help create these patterns. A watermarked LLM samples a token, with high probability, from the green list determined by its prefix token.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, through both empirical and theoretical analysis, we show that state-of-the-art AI-text detectors are not reliable in practical scenarios. We first study empirical attacks on soft watermarking [Kirchenbauer et al, 2023], and a wide range of zero-shot [Mitchell et al, 2023] and neural networkbased detectors [OpenAI, 2019]. We show that a paraphrasing attack, where a lightweight neural network-based paraphraser is applied to the output text of the AI-generative model, can evade various types of detectors.…”

Section: Introductionmentioning

confidence: 99%

“…Our experiments show that this automated paraphrasing attack can drastically reduce the accuracy of various detectors, including the ones using soft watermarking as well as neural network-based detectors and zero-shot classifiers. For example, a PEGASUS-based paraphraser [Zhang et al, 2019] can drop the soft watermarking detector's [Kirchenbauer et al, 2023] accuracy from 97% to 80% with just a degradation of 3.5 in the perplexity score. The area under the ROC curves of zero-shot detectors [Mitchell et al, 2023] drop from 96.5% to 25.2% using a T5-based paraphraser [Damodaran, 2021].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Can AI-Generated Text be Reliably Detected?

Sadasivan¹,

Kumar²,

Balasubramanian³

et al. 2023

Preprint

View full text Add to dashboard Cite

The rapid progress of Large Language Models (LLMs) has made them capable of performing astonishingly well on various tasks including document completion and question answering. The unregulated use of these models, however, can potentially lead to malicious consequences such as plagiarism, generating fake news, spamming, etc. Therefore, reliable detection of AI-generated text can be critical to ensure the responsible use of LLMs. Recent works attempt to tackle this problem either using certain model signatures present in the generated text outputs or by applying watermarking techniques that imprint specific patterns onto them. In this paper, both empirically and theoretically, we show that these detectors are not reliable in practical scenarios. Empirically, we show that paraphrasing attacks, where a light paraphraser is applied on top of the generative text model, can break a whole range of detectors, including the ones using the watermarking schemes as well as neural network-based detectors and zero-shot classifiers. We then provide a theoretical impossibility result indicating that for a sufficiently good language model, even the best-possible detector can only perform marginally better than a random classifier. Finally, we show that even LLMs protected by watermarking schemes can be vulnerable against spoofing attacks where adversarial humans can infer hidden watermarking signatures and add them to their generated text to be detected as text generated by the LLMs, potentially causing reputational damages to their developers. We believe these results can open an honest conversation in the community regarding the ethical and reliable use of AI-generated text.Preprint. Under review.

show abstract

Enhancing Creativity in Deep Learning Models with SAVE-Inspired Activation Functions

Brad

2023

IFIP Advances in Information and Communication Technology

View full text Add to dashboard Cite

A Watermark for Large Language Models

Cited by 44 publications

References 0 publications

ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education

ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education

Can AI-Generated Text be Reliably Detected?

Enhancing Creativity in Deep Learning Models with SAVE-Inspired Activation Functions

Contact Info

Product

Resources

About