Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering

Abstract

While machine learning (ML) models are being increasingly trusted to make decisions in different and varying areas, the safety of systems using such models has become an increasing concern. In particular, ML models are often trained on data from potentially untrustworthy sources, providing adversaries with the opportunity to manipulate them by inserting carefully crafted samples into the training set. Recent work has shown that this type of attack, called a poisoning attack, allows adversaries to insert backdoors or trojans into the model, enabling malicious behavior with simple external backdoor triggers at inference time and only a blackbox perspective of the model itself. Detecting this type of attack is challenging because the unexpected behavior occurs only when a backdoor trigger, which is known only to the adversary, is present. Model users, either direct users of training data or users of pre-trained model from a catalog, may not guarantee the safe operation of their ML-based system. In this paper, we propose a novel approach to backdoor detection and removal for neural networks. Through extensive experimental results, we demonstrate its effectiveness for neural networks classifying text and images. To the best of our knowledge, this is the first methodology capable of detecting poisonous data crafted to insert backdoors and repairing the model that does not require a verified and trusted dataset.

A General Framework for Defending Against Backdoor Attacks via Influence Graph

“…For the training-stage defense, a typical method is to leverage the spectral signatures to identify the malicious training examples (Tran et al, 2018;Hayase et al, 2021;Chen et al, 2018), based on the intuition that the representation of a poisoned example exposes a strong signal for the backdoor attack. Hong et al (2020) mitigates poison attacks by debiasing the gradient norm and length at each training step.…”

“…A common practice to find poisoned training data is to treat poisoned data points as outliers and apply outlier detection techniques. For example, Chen et al (2018) clusters intermediate representations (as we call representation-based method here), separating the poisonous from legitimate activations. Tran et al (2018) examines the spectrum of the covariance of a feature representation to detect the special spectrum signatures of malicious data points, gauged by the magnitude in the top PCA direction of that representation.…”

“…: the vanilla spectrum based method using PCA to detect poisoned data; (2) Clustering(Chen et al, 2018): clustering intermediate representations to separate the poisonous from legitimate data; (3) SPECTRA(Hayase et al, 2021): extending upon PCA by whitening the representations to amplify the spectrum signals; and (4) COSIN(Hammoudeh & Lowd, 2021): iteratively updating the influence of each data point using cached training checkpoints. The models are first trained on the hybrid dataset and the attack success rate Att suc is reported.…”

Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks

“…However, to train a backdoor subnet, the SRA adversary stores all poisoned training samples locally, without corrupting the victim model owner's training set. So all defenses utilizing the assumption that the training set being poisoned [9,10,14,61,64,65] are rendered ineffective.…”

Rank List Sensitivity of Recommender Systems to Interaction Perturbations

“…Such inadvertent presence of minimal unintentional perturbations should not impact the generated recommendations, but do they? Furthermore, in the worst-case, adversarial perturbations can be done by an attacker (e.g., hacker or company insider) who can access the training data [10,30,32]. What is the maximum damage that an attacker can do to the recommender system by introducing minimal adversarial perturbations?…”