Learning with Selective Forgetting

Abstract

Lifelong learning aims to train a highly expressive model for a new task while retaining all knowledge for previous tasks. However, many practical scenarios do not always require the system to remember all of the past knowledge. Instead, ethical considerations call for selective and proactive forgetting of undesirable knowledge in order to prevent privacy issues and data leakage. In this paper, we propose a new framework for lifelong learning, called Learning with Selective Forgetting, which is to update a model for the new task with forgetting only the selected classes of the previous tasks while maintaining the rest. The key is to introduce a class-specific synthetic signal called mnemonic code. The codes are "watermarked" on all the training samples of the corresponding classes when the model is updated for a new task. This enables us to forget arbitrary classes later by only using the mnemonic codes without using the original data. Experiments on common benchmark datasets demonstrate the remarkable superiority of the proposed method over several existing methods.

Towards Adversarial Evaluations for Inexact Machine Unlearning

“…Does not penalize better utility: We need evaluation methods to not penalize better utility. Consider removing a random subset of samples and computing error on the deletion set [27,35,56], with better unlearning models having a higher error. Such an evaluation can not differentiate between an unlearnt model that generalizes well to the (now unseen) deletion set versus one that does not unlearn at all.…”

“…The usefulness of a test is established by how reliably it quantifies the remaining information after applying any inexactunlearning method. We compare existing evaluation methods which use random sampling [27,35,39,45,51,56,61,68 5 and state consistent observations. Primarily, the IC test is the only test which reliably determines that exactly unlearning the final layer (1 layer-EU) does not remove generalized properties.…”

“…Inexact-unlearning: The primary goal of forgetting can often be hard to achieve, especially in deep networks. Hence, inexact-unlearning literature has relaxed the forgetting goal in two ways: not provable [10,32,35,39,45,56,67] and imperfect [10,31,32,35,39,51,65,68]. Relaxing provability implies unlearning methods do not provide any proven guarantees of information removal.…”

Continual Learning and Private Unlearning

“…The above methods all consider the MU problem in general where the preserved dataset (e.g., data except the removed ones) iis available, which is not the case in continual learning. Recently, a particularly relevant study first considers MU in the context of continual learning (Shibata et al, 2021). However, their problem definition aims to make the model predict as wrongly as possible on the removed data, which does not in general protect the user's privacy.…”

“…We call this novel problem continual learning and private unlearning (CLPU). To the best of our knowledge, only one previous paper discusses a similar problem setting pertaining to selective forgetting in continual learning (Shibata et al, 2021). However, the problem in that paper is different from CLPU as it defines forgetting as maximally degrading the performance on a task.…”