Measuring Forgetting of Memorized Training Examples

Abstract

Machine learning models exhibit two seemingly contradictory phenomena: training data memorization and various forms of forgetting. In memorization, models overfit specific training examples and become susceptible to privacy attacks. In forgetting, examples which appeared early in training are forgotten by the end. In this work, we connect these phenomena. We propose a technique to measure to what extent models "forget" the specifics of training examples, becoming less susceptible to privacy attacks on examples they have not seen recently. We show that, while non-convexity can prevent forgetting from happening in the worst-case, standard image and speech models empirically do forget examples over time. We identify nondeterminism as a potential explanation, showing that deterministically trained models do not forget. Our results suggest that examples seen early when training with extremely large datasets-for instance those examples used to pre-train a model-may observe privacy benefits at the expense of examples seen later.

Anatomy of Learned Database Tuning with Bayesian Optimization

“…Memorization in LLMs has been studied extensively as it comes with significant privacy and legal concerns (Carlini et al, 2021;2022b;Jagielski et al, 2022;Tirumala et al, 2022). Pretrained LLMs have a tendency to regurgitate verbatim some of the training data, and their memorization is proportional to their number of parameters (Carlini et al, 2021;Lee et al, 2021).…”

“…This memorization, albeit unintended, is crucial to model natural language, and for generalisation (Zhang et al, 2021b;Feldman, 2020). A range of metrics have been proposed to compute the level of memorization across data points in causal language modelling (Tirumala et al, 2022;Carlini et al, 2022b;Jagielski et al, 2022), based on the likelihood of emitting textual sequences verbatim.…”