Fast Yet Effective Machine Unlearning

Abstract

Unlearning the data observed during the training of a machine learning (ML) model is an important task that can play a pivotal role in fortifying the privacy and security of ML-based applications. This paper raises the following questions: (i) can we unlearn a class/classes of data from a ML model without looking at the full training data even once? (ii) can we make the process of unlearning fast and scalable to large datasets, and generalize it to different deep networks? We introduce a novel machine unlearning framework with error-maximizing noise generation and impair-repair based weight manipulation that offers an efficient solution to the above questions. An error-maximizing noise matrix is learned for the class to be unlearned using the original model. The noise matrix is used to manipulate the model weights to unlearn the targeted class of data. We introduce impair and repair steps for a controlled manipulation of the network weights. In the impair step, the noise matrix along with a very high learning rate is used to induce sharp unlearning in the model. Thereafter, the repair step is used to regain the overall performance. With very few update steps, we show excellent unlearning while substantially retaining the overall model accuracy. Unlearning multiple classes requires a similar number of update steps as for the single class, making our approach scalable to large problems. Our method is quite efficient in comparison to the existing methods, works for multi-class unlearning, doesn't put any constraints on the original optimization mechanism or network design, and works well in both small and large-scale vision tasks. This work is an important step towards fast and easy implementation of unlearning in deep networks. We will make the source code publicly available.

Certifiable Unlearning Pipelines for Logistic Regression: An Experimental Study

“…One direction is to extend the study to methods for more complex models. While the proposed unlearning pipeline can be extended to non-linear ML models such as neural networks with appropriate unlearning methods [37][38][39], extending the experimental evaluation (Section 6) is non-trivial. This is due to the stochasticity of learning [11] for non-linear ML models, which results in many local minima (i.e., several equally valid, fully retrained models with different accuracies).…”

Feature Unlearning for Pre-trained GANs and VAEs

“…Unlearning scenarios can vary depending on the requirements (Nguyen et al 2022). Traditional machine unlearning approaches assume that all training data can be accessed (Gupta et al 2021;Tarun et al 2021;Baumhauer, Schöttle, and Zeppelzauer 2022;Ginart et al 2019). However, recent studies have presented problem formulations in which access to the data is highly restricted (Yoon et al 2022;Chundawat et al 2022;Golatkar, Achille, and Soatto 2020;Nguyen, Low, and Jaillet 2020).…”

“…Machine unlearning aims to erase the target data from a pre-trained machine-learning model, which can be required to remove private information, harmful content, and biased information (Cao and Yang 2015). However, most of the machine unlearning methods have been focused on supervised models so far (Gupta et al 2021;Tarun et al 2021;Baumhauer, Schöttle, and Zeppelzauer 2022;Ginart et al 2019;Yoon et al 2022;Chundawat et al 2022;Golatkar, Achille, and Soatto 2020;Nguyen, Low, and Jaillet 2020).…”

Audit to Forget: A Unified Method to Revoke Patients’ Private Data in Intelligent Healthcare

“…Model-intrinsic approaches are those methods designed for specific types of models, such as for softmax classifiers [30], linear models [31], treebased models [32] and Bayesian models [19]. Data-driven approaches focus on the data itself, including data partitioning [18], data augmentation [33], [34], [35] and other unlearning strategies based on data influence [36]. All methods have their specific application scenarios and limitations.…”