Knowledge Cross-Distillation for Membership Privacy

Abstract

A membership inference attack (MIA) poses privacy risks for the training data of a machine learning model. With an MIA, an attacker guesses if the target data are a member of the training dataset. The state-of-the-art defense against MIAs, distillation for membership privacy (DMP), requires not only private data for protection but a large amount of unlabeled public data. However, in certain privacy-sensitive domains, such as medicine and finance, the availability of public data is not guaranteed. Moreover, a trivial method for generating public data by using generative adversarial networks significantly decreases the model accuracy, as reported by the authors of DMP. To overcome this problem, we propose a novel defense against MIAs that uses knowledge distillation without requiring public data. Our experiments show that the privacy protection and accuracy of our defense are comparable to those of DMP for the benchmark tabular datasets used in MIA research, Purchase100 and Texas100, and our defense has a much better privacy-utility trade-off than those of the existing defenses that also do not use public data for the image dataset CIFAR10.

SEDMA: Self-Distillation with Model Aggregation for Membership Privacy

“…privacy, model utility, and computational costs, compared to stateof-the-art empirical defenses, such as AdvReg [23], MemGuard [16], KCD [8], and SELENA [38].…”

“…Our goal is to mitigate practical black-box MIAs, maintain high model utility, and have low computational costs. First, similar to KCD [8], SEDMA splits a training dataset into several subsets and trains multiple ML models (called sub-models) on each subset. Second, the trained sub-models are aggregated into several pairs (called model aggregation).…”

“…This process is called self-distillation, transferring prediction vectors (knowledge distillation) between ML models with the same model architecture. The state-of-the-art MIA defenses, KCD [8] and SELENA [38], have achieved favorable privacy-utility trade-offs by self-distillation on split training datasets. Note that the novelty of SEDMA lies in the approach of applying model aggregation in several pairs of sub-models for self-distillation.…”

“…In our experiments, we evaluated SEDMA on three benchmark datasets (Purchase100, Texas100, and CIFAR100). We compared SEDMA with four existing empirical MIA defenses [8,16,23,38], including KCD and SELENA. We conducted two types of black-box MIAs, single-query attacks (NN-based attacks and metric-based 1 SElf-Distillation using Model Aggregation attacks) and label-only attacks (boundary distance attacks, data augmentation attacks, and likelihood ratio attacks) as existing attacks that have been used for evaluation in related work.…”

“…MIA defenses need to design ML models to behave similarly between a sample of members and non-members because MIAs identify the sample as members or non-members based on the different model behavior caused by whether or not the sample is included in the training data [1,8,14,16,23,32,38]. MIA defenses can be categorized into provable privacy defenses and empirical membership privacy defenses, as shown in Table 1.…”