Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks

Umer, Muhammad; Dawson, Glenn; Polikar, Robi

doi:10.1109/ijcnn48605.2020.9206809

Cited by 11 publications

(6 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Fisher information on the pre-trained model is considered as the cause of this problem; it contains information to be forgotten. We believe that the reason for the insufficient accuracy is that the effects of the information to be forgotten contained in Fisher information hardly disappear in EWC (Umer, Dawson, and Polikar 2020).…”

Section: Discussionmentioning

confidence: 95%

Selective Forgetting of Deep Networks at a Finer Level than Samples

Hayase¹,

Yasutomi²,

Katoh³

2020

Preprint

View full text Add to dashboard Cite

Selective forgetting or removing information from deep neural networks (DNNs) is essential for continual learning and is challenging in controlling the DNNs. Such forgetting is crucial also in a practical sense since the deployed DNNs may be trained on the data with outliers, poisoned by attackers, or with leaked/sensitive information. In this paper, we formulate selective forgetting for classification tasks at a finer level than the samples' level. We specify the finer level based on four datasets distinguished by two conditions: whether they contain information to be forgotten and whether they are available for the forgetting procedure. Additionally, we reveal the need for such formulation with the datasets by showing concrete and practical situations. Moreover, we introduce the forgetting procedure as an optimization problem on three criteria; the forgetting, the correction, and the remembering term. Experimental results show that the proposed methods can make the model forget to use specific information for classification. Notably, in specific cases, our methods improved the model's accuracy on the datasets, which contains information to be forgotten but is unavailable in the forgetting procedure. Such data are unexpectedly found and misclassified in actual situations.

show abstract

Section: Discussionmentioning

confidence: 95%

Selective Forgetting of Deep Networks at a Finer Level than Samples

Hayase¹,

Yasutomi²,

Katoh³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In the literature, researchers have used incremental learning techniques to detect fake news using Artificial Neural Networks (ANN). However, ANNs suffer from catastrophic forgetting which lowers the performance of the model as data streams arrive [20]. The deep learning and neural network-based techniques can classify short text appearing sequentially but require large memory space and training time, thus reducing the performance of the model [19].…”

Section: A Incremental Learning Approachmentioning

confidence: 99%

Detecting and Fact-checking Misinformation using “Veracity Scanning Model”

Barve¹,

Saini²,

Kotecha³

et al. 2022

IJACSA

View full text Add to dashboard Cite

The expeditious flow of information over the web and its ease of convenience has increased the fear of the rampant spread of misinformation. This poses a health threat and an unprecedented issue to the world impacting people's life. To cater to this problem, there is a need to detect misinformation. Recent techniques in this area focus on static models based on feature extraction and classification. However, data may change at different time intervals and the veracity of data needs to be checked as it gets updated. There is a lack of models in the literature that can handle incremental data, check the veracity of data and detect misinformation. To fill this gap, authors have proposed a novel Veracity Scanning Model (VSM) to detect misinformation in the healthcare domain by iteratively factchecking the contents evolving over the period of time. In this approach, the healthcare web URLs are classified as legitimate or non-legitimate using sentiment analysis as a feature, document similarity measures to perform fact-checking of URLs, and incremental learning to handle the arrival of incremental data. The experimental results show that the Jaccard Distance measure has outperformed other techniques with an accuracy of 79.2% with Random Forest classifier while the Cosine similarity measure showed less accuracy of 60.4% with the Support Vector Machine classifier. Also, when implemented as an algorithm Euclidean distance showed an accuracy of 97.14% and 98.33% respectively for train and test data.

show abstract

“…Both regularization-and generative replay-based CL approaches work reasonably well in retaining prior knowledge, but their vulnerability to adversarial attacks has only recently started to be explored. For example, the vulnerability of EWC to perceptible backdoor attacks was investigated in [13], and the vulnerability of related importance based domain adaptation approaches to optimization based poisoning attacks has been discussed in our prior work [14], [15]. However, to the best of our knowledge, the vulnerabilities and robustness of other regularization-based approaches, as well as those of the more successful and robust generative replay-based approaches [5] have not yet been determined.…”

Section: A Continual Learningmentioning

confidence: 99%

“…In our prior work, we showed the vulnerability of a specific regularization-based continual learning algorithm, elastic weight consolidation, to adversarial backdoor poisoning attacks with visibly obvious backdoor patterns (such as small subimages) that are embedded into the training dataset with a false label of the attacker's choosing [6]. In this effort, we extend that work to demonstrate the same vulnerability in other state-of-the-art regularization-based and generative replay-based CL algorithms.…”

Section: Introductionmentioning

confidence: 99%

Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models

Umer¹,

Polikar²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Continual (or "incremental") learning approaches are employed when additional knowledge or tasks need to be learned from subsequent batches or from streaming data. However these approaches are typically adversary agnostic, i.e., they do not consider the possibility of a malicious attack. In our prior work, we explored the vulnerabilities of Elastic Weight Consolidation (EWC) to the perceptible misinformation. We now explore the vulnerabilities of other regularization-based as well as generative replay-based continual learning algorithms, and also extend the attack to imperceptible misinformation. We show that an intelligent adversary can take advantage of a continual learning algorithm's capabilities of retaining existing knowledge over time, and force it to learn and retain deliberately introduced misinformation. To demonstrate this vulnerability, we inject backdoor attack samples into the training data. These attack samples constitute the misinformation, allowing the attacker to capture control of the model at test time. We evaluate the extent of this vulnerability on both rotated and split benchmark variants of the MNIST dataset under two important domain and class incremental learning scenarios. We show that the adversary can create a "false memory" about any task by inserting carefullydesigned backdoor samples to the test instances of that task thereby controlling the amount of forgetting of any task of its choosing. Perhaps most importantly, we show this vulnerability to be very acute and damaging: the model memory can be easily compromised with the addition of backdoor samples into as little as 1% of the training data, even when the misinformation is imperceptible to human eye.

show abstract

Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks

Cited by 11 publications

References 29 publications

Selective Forgetting of Deep Networks at a Finer Level than Samples

Selective Forgetting of Deep Networks at a Finer Level than Samples

Detecting and Fact-checking Misinformation using “Veracity Scanning Model”

Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models

Contact Info

Product

Resources

About