Coded Machine Unlearning

Abstract

Models trained in machine learning processes may store information about individual samples used in the training process. There are many cases where the impact of an individual sample may need to be deleted and unlearned (i.e., removed) from the model. Retraining the model from scratch after removing a sample from its training set guarantees perfect unlearning, however, it becomes increasingly expensive as the size of training dataset increases. One solution to this issue is utilizing an ensemble learning method that splits the dataset into disjoint shards and assigns them to non-communicating weak learners and then aggregates their models using a pre-defined rule. This framework introduces a trade-off between performance and unlearning cost which may result in an unreasonable performance degradation, especially as the number of shards increases. In this paper, we present a coded learning protocol where the dataset is linearly coded before the learning phase. We also present the corresponding unlearning protocol for the aforementioned coded learning model along with a discussion on the proposed protocol's success in ensuring perfect unlearning. Finally, experimental results show the effectiveness of the coded machine unlearning protocol in terms of performance versus unlearning cost trade-off.INDEX TERMS Machine unlearning, random coding, efficient sample removal.

Towards Adversarial Evaluations for Inexact Machine Unlearning

“…This ensures architecturally [6,11,53] or temporally [11,35] isolating the influence of any sample to a limited part of training, requiring retraining for only the affected parts. Isolation has been used across techniques like Linear Classification [6], Random Forest [13,54], KNN [1], SVM [16,63] and DNN [11,30,35] by utilizing or creating a sparse influence graph [53]. Datainfluence isolation often comes at the cost of utility as each portion becomes a weaker learner [8], especially in deep networks [58].…”

“…Isolation-based strategies change the training process by creating an ensemble [11,30,35,53], each of whose models is trained on different subsets of the dataset. This ensures architecturally [6,11,53] or temporally [11,35] isolating the influence of any sample to a limited part of training, requiring retraining for only the affected parts. Isolation has been used across techniques like Linear Classification [6], Random Forest [13,54], KNN [1], SVM [16,63] and DNN [11,30,35] by utilizing or creating a sparse influence graph [53].…”

“…• Dataset D M IA is created with the probability outputs for class t: M (D f ) t and M (D u ) t stored as class 0 and class 1 respectively. • We then create a 50-50 6 shadow (D M IA−S ) -test (D M IA−T ) split of D M IA . • A threshold p t needs to be chosen such that probabilities > p t are classified as class 0, and probabilities < p t as class 1.…”

Machine Unlearning of Features and Labels

“…In this setting, the unlearning of data points can be efficiently carried out by only retraining the affected submodels. Aldaghri et al [3] show that this approach can be further sped up for least-squares regression by choosing the shards cleverly. Unlearning based on shards, however, is suitable for removing a few data points only and inevitably deteriorates in performance when larger portions of the data require changes.…”

Evaluating Machine Unlearning via Epistemic Uncertainty

“…Several approaches were presented, which apply to different models and come with different assumptions and restrictions. So far, these approaches cover unlearning in decision trees and random forests [3,22], linear models such as logistic regression [1,8,10], neural networks [2,7,8,9,10] and even Markov Chain Monte Carlo [6,19]. While most approaches focus on forgetting 1 in a single model, there also exist works that deal with federated models instead [29].…”