Understanding Membership Inferences on Well-Generalized Learning Models

Abstract

Membership Inference Attack (MIA) determines the presence of a record in a machine learning model's training data by querying the model. Prior work has shown that the attack is feasible when the model is overfitted to its training data or when the adversary controls the training algorithm. However, when the model is not overfitted and the adversary does not control the training algorithm, the threat is not well understood. In this paper, we report a study that discovers overfitting to be a sufficient but not a necessary condition for an MIA to succeed. More specifically, we demonstrate that even a well-generalized model contains vulnerable instances subject to a new generalized MIA (GMIA). In GMIA, we use novel techniques for selecting vulnerable instances and detecting their subtle influences ignored by overfitting metrics. Specifically, we successfully identify individual records with high precision in real-world datasets by querying black-box machine learning models. Further we show that a vulnerable record can even be indirectly attacked by querying other related records and existing generalization techniques are found to be less effective in protecting the vulnerable instances. Our findings sharpen the understanding of the fundamental cause of the problem: the unique influences the training instance may have on the model.

On the Importance of Difficulty Calibration in Membership Inference Attacks

“…We argue that a simple modification to the score, which we call difficulty calibration, can drastically improve the attack's reliability. This approach has been applied to the loss score for high-precision attack against well-generalized models (Long et al, 2018). Let s(h, (x, y)) be the membership score, where higher score indicates a stronger signal that the sample (x, y) is a member.…”

“…Various forms of difficulty calibration have been considered in the context of highprecision attacks. Long et al (2018) selected samples that differ the most in loss between the target and a set of reference models, and showed that the resulting attack has high precision even for well-generalized target models. Carlini et al (2020) showed that such privacy attacks are also possible on large-scale language models such as GPT-2 (Radford et al, 2019).…”

Enhanced Membership Inference Attacks against Machine Learning Models

“…A substantial body of literature followed this work extending the attacks to different setting such as white box analysis [Nasr et al, 2019, Sablayrolles et al, 2019, Leino and Fredrikson, 2020, label-only access [Li andZhang, 2020, Choquette-Choo et al, 2021], federated learning [Nasr et al, 2019, transfer learning [Zou et al, 2020] and different types of data, models such as aggregate location data [Pyrgelis et al, 2017], generative models [Hayes et al, 2019], language models [Carlini et al, 2019, Carlini et al, 2020, sentence embeddings [Song and Raghunathan, 2020], and speech recognition models [Shah et al, 2021]. Multiple works have looked at improving the attack methodology through a more fine grained analysis or by reducing the background knowledge and the compute power required to execute the attack [Long et al, 2018, Song and Mittal, 2021, Salem et al, 2018. All these works follow the same attack framework for membership inference, but they either exploit a slightly different signal that is correlated with membership of a point in the training set or find an efficient way to exploit the already known signals.…”

“…4.1 Attack S: MIA via Shadow modelsStarting fromShokri et al [2017], a substantial body of literatureNasr et al [2019],Sablayrolles et al [2019],Leino and Fredrikson [2020],Long et al [2018],Song and Mittal [2021],Salem et al …”

Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture

“…They formalize the attack as a binary classification task and utilize a neural network (NN) model along with shadow training technique to distinguish members of training set from non-members. Following this work, many MIAs have been proposed which can be divided into two categories: direct attacks [32,33,46,47,54,55], which directly query the target sample and typically utilize only a single query; indirect attacks [8,28,29], which query for samples that are in the neighborhood of the target sample to infer membership, and typically utilize multiple queries. The research community has further extended the MIA to federated settings [31,33] and generative models [14].…”