Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

Tirumala, Kushal; Markosyan, Aram; Zettlemoyer, Luke; Aghajanyan, Armen

doi:10.48550/arxiv.2205.10770

Cited by 13 publications

(12 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent study shows that large language models can memorize its training data, and generate texts from training data given certain prompts Kharitonov et al, 2021;Thakkar et al, 2020;Carlini et al, 2019;Tirumala et al, 2022). Most related to our work, Carlini et al (2022) found that the memorization ability of LLMs significantly grows as the model capacity increases, the number of times an example has been duplicated, and the number of tokens of context used to prompt the model.…”

Section: Memorization In Large Language Modelssupporting

confidence: 56%

Recitation-Augmented Language Models

Sun¹,

Wang²,

Tay³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a new paradigm to help Large Language Models (LLMs) generate more accurate factual knowledge without retrieving from an external corpus, called RECITation-augmented gEneration (RECITE). Different from retrievalaugmented language models that retrieve relevant documents before generating the outputs, given an input, RECITE first recites one or several relevant passages from LLMs' own memory via sampling, and then produces the final answers. We show that RECITE is a powerful paradigm for knowledge-intensive NLP tasks. Specifically, we show that by utilizing recitation as the intermediate step, a recite-and-answer scheme can achieve new state-of-the-art performance in various closed-book question answering (CBQA) tasks. In experiments, we verify the effectiveness of RECITE on three pre-trained models (PaLM, UL2, and OPT) and three CBQA tasks (Natural Questions, TriviaQA, and HotpotQA). Question: who wrote the song i hate you i love you Answer: Gnash … Question: who wrote the school for good and evil Direct Generation (e.g., PaLM) Answer: Soman Chainani LM Input LM Output Question: who wrote the school for good and evil Retrieval-augmented Generation (e.g., Atlas) Answer: Soman Chainani LM Input LM Output The School for Good and Evil is a fantasy fairytale hexalogy of books by Soman Chainani… Retriever Results Question: who wrote the song i hate you i love you Recitation: "I Hate U, I Love U" (stylized in all lowercase) is a song by American singer and rapper Gnash featuring American singer Olivia O'Brien. Answer: Gnash … Question: who wrote the school for good and evil Recitation-augmented Generation (ours) Recitation: The School for Good and Evil was first published on May 14, 2013 by Soman Chainani… Answer: Soman Chainani LM Input LM Output * Work done during internship at Google.

show abstract

Section: Memorization In Large Language Modelssupporting

confidence: 56%

Recitation-Augmented Language Models

Sun¹,

Wang²,

Tay³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…MA quantifies how much f θ has memorized the given token sequences and was proposed by Tirumala et al (2022) to analyze the training dynamics of large LMs.…”

Section: Extraction Likelihood (El)mentioning

confidence: 99%

Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Jang¹,

Yoon²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply applying the unlikelihood training objective to target token sequences is effective at forgetting them with little to no degradation of general language modeling performances; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being orders of magnitude more computationally efficient. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning.

show abstract

“…This definition and variations of it have been used widely in the literature (Kandpal et al, 2022;Lee et al, 2021;Carlini et al, 2022). For example, Tirumala et al (2022) study a similar per-token definition called exact memorization and Kandpal et al (2022) a document-level definition called perfect memorization.…”

Section: Measuring Verbatim Memorizationmentioning

confidence: 99%

Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

Ippolito¹,

Tramèr²,

Nasr³

et al. 2022

Preprint

View full text Add to dashboard Cite

Studying data memorization in neural language models helps us understand the risks (e.g., to privacy or copyright) associated with models regurgitating training data, and aids in the evaluation of potential countermeasures. Many prior works-and some recently deployed defenses-focus on "verbatim memorization", defined as a model generation that exactly matches a substring from the training set. We argue that verbatim memorization definitions are too restrictive and fail to capture more subtle forms of memorization. Specifically, we design and implement an efficient defense based on Bloom filters that perfectly prevents all verbatim memorization. And yet, we demonstrate that this "perfect" filter does not prevent the leakage of training data. Indeed, it is easily circumvented by plausible and minimally modified "style-transfer" prompts-and in some cases even the non-modified original prompts-to extract memorized information. For example, instructing the model to output ALL-CAPITAL texts bypasses memorization checks based on verbatim matching. We conclude by discussing potential alternative definitions and why defining memorization is a difficult yet crucial open question for neural language models.˚Remaining authors ordered by Algorithm 15 in Appendix I; briefly, we require Daphne be listed first, and Nicholas listed last, and we search for the first permutation of authors' first names which satisfies these constraints, where permutations order names by their salted MD5 hash.

show abstract

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

Cited by 13 publications

References 51 publications

Recitation-Augmented Language Models

Recitation-Augmented Language Models

Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

Contact Info

Product

Resources

About