Federated Unlearning via Class-Discriminative Pruning

Abstract

We explore the problem of selectively forgetting categories from trained CNN classification models in the federated learning (FL). Given that the data used for training cannot be accessed globally in FL, our insights probe deep into the internal influence of each channel. Through the visualization of feature maps activated by different channels, we observe that different channels have a varying contribution to different categories in image classification. Inspired by this, we propose a method for scrubbing the model clean of information about particular categories. The method does not require retraining from scratch, nor global access to the data used for training. Instead, we introduce the concept of Term Frequency Inverse Document Frequency (TF-IDF) to quantize the class discrimination of channels. Channels with high TF-IDF scores have more discrimination on the target categories and thus need to be pruned to unlearn. The channel pruning is followed by a fine-tuning process to recover the performance of the pruned model. Evaluated on CIFAR10 dataset, our method accelerates the speed of unlearning by 8.9× for the ResNet model, and 7.9× for the VGG model under no degradation in accuracy, compared to retraining from scratch. For CIFAR100 dataset, the speedups are 9.9× and 8.4×, respectively. We envision this work as a complementary block for FL towards compliance with legal and ethical criteria.

Evaluating Machine Unlearning via Epistemic Uncertainty

“…In many works [1,2,6,8,10,11,18,29], retraining the model from scratch on the remaining data D r is considered the optimal solution, since it does not use the sensitive data points D f during training and achieves high performance on D r . However, for large models and datasets retraining from scratch is computationally expensive, which is why this is often considered impractical.…”

“…These three aspects form the foundation for evaluating Machine Unlearning algorithms and are widely agreed on as they are stated explicitly or implicitly in many works in the domain [2,3,4,6,7,8,9,22,24,27,29]. Here, we make use of the terminology as stated by Warnecke et al in [30].…”

“…So far, these approaches cover unlearning in decision trees and random forests [3,22], linear models such as logistic regression [1,8,10], neural networks [2,7,8,9,10] and even Markov Chain Monte Carlo [6,19]. While most approaches focus on forgetting 1 in a single model, there also exist works that deal with federated models instead [29]. The vast majority of Machine Unlearning algorithms are motivated by legal requirements such as the General Data Protection Regulation (GDPR) or the California Consumer Privacy Act.…”

“…Finally, we have those approaches that are neither perfect nor give guarantees [7,8,9,16,22,28,29], but evaluate the success of unlearning purely empirical or compare the resulting model with an actual retrained model. The latter might be interesting from a theoretical point of view but is inapplicable in practice.…”

Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher

“…Existing work in this area, [7,8,9,11,12] show effectiveness only for image classification. There is a need for a generic unlearning method that can work across different applications domains.…”