Learning with Recoverable Forgetting

Abstract

Life-long learning aims at learning a sequence of tasks without forgetting the previously acquired knowledge. However, the involved training data may not be life-long legitimate due to privacy or copyright reasons. In practical scenarios, for instance, the model owner may wish to enable or disable the knowledge of specific tasks or specific samples from time to time. Such flexible control over knowledge transfer, unfortunately, has been largely overlooked in previous incremental or decremental learning methods, even at a problem-setup level. In this paper, we explore a novel learning scheme, termed as Learning wIth Recoverable Forgetting (LIRF), that explicitly handles the task-or sample-specific knowledge removal and recovery. Specifically, LIRF brings in two innovative schemes, namely knowledge deposit and withdrawal, which allow for isolating user-designated knowledge from a pre-trained network and injecting it back when necessary. During the knowledge deposit process, the specified knowledge is extracted from the target network and stored in a deposit module, while the insensitive or general knowledge of the target network is preserved and further augmented. During knowledge withdrawal, the taken-off knowledge is added back to the target network. The deposit and withdraw processes only demand for a few epochs of finetuning on the removal data, ensuring both data and time efficiency. We conduct experiments on several datasets, and deomnstrate that the proposed LIRF strategy yields encouraging results with gratifying generalization capability.

Learning to Unlearn: Instance-Wise Unlearning for Pre-trained Classifiers

“…Recent work have studied machine unlearning for deep neural networks in various setups, which can be categorized into two approaches: class-wise and instance-wise unlearning. Class-wise unlearning forgets all data points belonging to a certain class (e.g., all images of dogs in CIFAR-10) while retaining performance on the remaining classes (Tarun et al 2021;Chundawat et al 2022;Ye et al 2022;Yoon et al 2022;Graves, Nagisetty, and Ganesh 2021). In contrast, instance-wise unlearning deletes information from individual data points with mixed classes (Golatkar, Achille, and Soatto 2020; Kim and Woo 2022;Mehta et al 2022).…”

“…While several machine unlearning approaches have shown promising results deleting data from traditional machine learning algorithms (Mahadevan and Mathioudakis 2021;Ginart et al 2019;Brophy and Lowd 2021) as well as DNN-based classifiers (Tarun et al 2021;Chundawat et al 2022;Ye et al 2022;Yoon et al 2022;Golatkar, Achille, and Soatto 2020;Kim and Woo 2022;Mehta et al 2022), existing work are built upon assumptions far too restrictive compared to real-life scenarios. First off, many approaches assume a class-wise unlearning setup, where the task is to delete information from all data points that belong to a particular class or set of classes.…”