Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models

Abstract

Continual (or "incremental") learning approaches are employed when additional knowledge or tasks need to be learned from subsequent batches or from streaming data. However these approaches are typically adversary agnostic, i.e., they do not consider the possibility of a malicious attack. In our prior work, we explored the vulnerabilities of Elastic Weight Consolidation (EWC) to the perceptible misinformation. We now explore the vulnerabilities of other regularization-based as well as generative replay-based continual learning algorithms, and also extend the attack to imperceptible misinformation. We show that an intelligent adversary can take advantage of a continual learning algorithm's capabilities of retaining existing knowledge over time, and force it to learn and retain deliberately introduced misinformation. To demonstrate this vulnerability, we inject backdoor attack samples into the training data. These attack samples constitute the misinformation, allowing the attacker to capture control of the model at test time. We evaluate the extent of this vulnerability on both rotated and split benchmark variants of the MNIST dataset under two important domain and class incremental learning scenarios. We show that the adversary can create a "false memory" about any task by inserting carefullydesigned backdoor samples to the test instances of that task thereby controlling the amount of forgetting of any task of its choosing. Perhaps most importantly, we show this vulnerability to be very acute and damaging: the model memory can be easily compromised with the addition of backdoor samples into as little as 1% of the training data, even when the misinformation is imperceptible to human eye.

Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks

“…Without any loss of generality, we assume that the target task is Task 1. To degrade the test time performance of the target task, the attacker can insert malicious samples into the training data of all non-target tasks (i.e., all tasks except Task 1 as done in our prior work [18]. However, in order to make the attack even more difficult to detect, we now restrict the attacker to add small amount of imperceptible misinformation only at the last task to degrade the test time performance of the first task.…”

“…In preliminary stages of this work, we discussed vulnerability of importance based domain adaptation to poisoning attacks in [16], [17], the vulnerability of EWC under task incremental setting to perceptible backdoor attacks in [7], and the vulnerabilities of regularization and generative replay approaches considering only the simple MNIST dataset and where the attack was launched on all tasks in [18]. This effort significantly expands the prior work to not only the more challenging SVHN and CIFAR10 datasets, but also to a more realistic scenario where the attacker inserts imperceptible misinformation to only a single task (as opposed to all tasks in prior work) of its own choosing.…”

“…Two major types of such attacks are [19]: i) causative (or poisoning) attacks, which add strategically-chosen malicious data points into the training data [16], [17], [20] to impair future generalization capabilities of the classifier; and ii) exploratory (or evasion) attacks, which exploit misclassification at the test time to discover a model's blind spots or to extract information about the data or the model itself [21]. Since CL involves retraining / updating the model with each new data, the adversary's choice in targeting CL algorithms is typically a poisoning attack [7], [18].…”

Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models

“…While several CL approaches have been proposed to avoid the problem of catastrophic forgetting, but recently it has been found that these approaches are extremely vulnerable to adversarial backdoor attacks [3]- [5], where an intelligent adversary can easily insert miniature amount of misinformation in the training data to deliberately or intentionally disturb the balance between stability and plasticity acquired by the CL model. More specifically, the goal of such an attack is to artificially increase the forgetting of the CL model on a explicitly targeted previously learned task.…”