Reconstructing Training Data with Informed Adversaries

Abstract

Given access to a machine learning model, can an adversary reconstruct the model's training data? This work studies this question from the lens of a powerful informed adversary who knows all the training data points except one. By instantiating concrete attacks, we show it is feasible to reconstruct the remaining data point in this stringent threat model. For convex models (e.g. logistic regression), reconstruction attacks are simple and can be derived in closed-form. For more general models (e.g. neural networks), we propose an attack strategy based on training a reconstructor network that receives as input the weights of the model under attack and produces as output the target data point. We demonstrate the effectiveness of our attack on image classifiers trained on MNIST and CIFAR-10, and systematically investigate which factors of standard machine learning pipelines affect reconstruction success. Finally, we theoretically investigate what amount of differential privacy suffices to mitigate reconstruction attacks by informed adversaries. Our work provides an effective reconstruction attack that model developers can use to assess memorization of individual points in general settings beyond those considered in previous works (e.g. generative language models or access to training gradients); it shows that standard models have the capacity to store enough information to enable high-fidelity reconstruction of training data points; and it demonstrates that differential privacy can successfully mitigate such attacks in a parameter regime where utility degradation is minimal.

Bayes Security: A Not So Average Metric

“…Overall, we expect future work can provide Bayes security-style guarantees for complex ML training pipelines. For example, by exploiting our results on the Gaussian mechanism (Section VI), it may be possible to study the security of DP-SGD against common attacks such as membership inference [31], attribute inference [20], and reconstruction [4], [7]. This will enable bypassing bounds relating ε and the advantage [22], [40], by computing the advantage (or Bayes security) directly.…”

“…One of the most widely used vulnerability metrics is Bayes vulnerability [32], defined as V (π) = max s π s = 1 − G(π); it expresses the adversary's probability of guessing the secret correctly in one try. 4 For the posterior version, it holds that V (π, C) = 1−R * (π, C). The multiplicative risk leakage follows the same core idea: G(π) can be thought of as a prior version of R * : indeed, it holds that R * (π, C) = o p(o)G(δ o ) where δ o are the posteriors of the channel.…”

Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

“…These works also recover "representative" images from different classes, rather than specific training examples. Recent work on reconstructing training images have used feature similarity (Haim et al, 2022) and pixel similarity (Balle et al, 2022). In each of these papers, "fuzzy" reconstructions are allowed by the evaluation metrics and, indeed, are common in their reconstructions.…”

CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learning

“…For example, membership inference attacks (MIA) attempt to distinguish whether a sample was present in the training set given only the trained model (Shokri et al, 2017;Sablayrolles et al, 2019). Others attacks consider the more difficult problem of reconstructing entire training samples from a trained model, often using (batch) gradient information Jeon et al, 2021;Balle et al, 2022). Since model updates u i are essentially just aggregated gradients, it is natural that FL updates may leak private information as well.…”