Concealed Data Poisoning Attacks on NLP Models

Wallace, Eric; Zhao, Tony Z.; Feng, Shi; Singh, Sameer

doi:10.18653/v1/2021.naacl-main.13

Cited by 59 publications

(40 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these cannot be directly applied to recommender systems due to sequential data dependency. Recently, data perturbation attacks [9,34,54] for natural language processing (NLP) have been proposed. However, we cannot employ them directly for our setting since they either are targeted perturbations, have different perturbation levels (e.g., word or embedding modifications), or cannot model the long sequential dependency.…”

Section: Related Workmentioning

confidence: 99%

Rank List Sensitivity of Recommender Systems to Interaction Perturbations

Oh¹,

Kumar²

2022

Preprint

View full text Add to dashboard Cite

While deep learning-based sequential recommender systems are widely used in practice, their sensitivity to untargeted training data perturbations is unknown. Untargeted perturbations aim to modify ranked recommendation lists for all users at test time, by inserting imperceptible input perturbations during training time. Existing perturbation methods are mostly targeted attacks optimized to change ranks of target items, but not suitable for untargeted scenarios. In this paper, we develop a novel framework in which user-item training interactions are perturbed in unintentional and adversarial settings. First, through comprehensive experiments on four datasets, we show that four popular recommender models are unstable against even one random perturbation. Second, we establish a cascading effect in which minor manipulations of early training interactions can cause extensive changes to the model and the generated recommendations for all users. Leveraging this effect, we propose an adversarial perturbation method CASPER which identifies and perturbs an interaction that induces the maximal cascading effect. Experimentally, we demonstrate that CASPER reduces the stability of recommendation models the most, compared to several baselines and state-of-the-art methods. Finally, we show the runtime and success of CASPER scale near-linearly with the dataset size and the number of perturbations, respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

Rank List Sensitivity of Recommender Systems to Interaction Perturbations

Oh¹,

Kumar²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, these carefully designed samples can be identified as outliers and filtered [41]. Another popular data poisoning way is generating adversarial samples to subvert the training process [44]. For example, Yang et al [53] proposed to use autoencoder as the generator to create poisoned data, which is further updated based on a reward function of loss.…”

Section: Related Workmentioning

confidence: 99%

FedAttack: Effective and Covert Poisoning Attack on Federated Recommendation via Hard Sampling

Wu¹,

Wu²,

Qi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is a feasible technique to learn personalized recommendation models from decentralized user data. Unfortunately, federated recommender systems are vulnerable to poisoning attacks by malicious clients. Existing recommender system poisoning methods mainly focus on promoting the recommendation chances of target items due to financial incentives. In fact, in real-world scenarios, the attacker may also attempt to degrade the overall performance of recommender systems. However, existing general FL poisoning methods for degrading model performance are either ineffective or not concealed in poisoning federated recommender systems. In this paper, we propose a simple yet effective and covert poisoning attack method on federated recommendation, named FedAttack. Its core idea is using globally hardest samples to subvert model training. More specifically, the malicious clients first infer user embeddings based on local user profiles. Next, they choose the candidate items that are most relevant to the user embeddings as hardest negative samples, and find the candidates farthest from the user embeddings as hardest positive samples. The model gradients inferred from these poisoned samples are then uploaded to the server for aggregation and model update. Since the behaviors of malicious clients are somewhat similar to users with diverse interests, they cannot be effectively distinguished from normal clients by the server. Extensive experiments on two benchmark datasets show that FedAttack can effectively degrade the performance of various federated recommender systems, meanwhile cannot be effectively detected nor defended by many existing methods.

show abstract

“…Data poisoning attacks only perturb adversarial set D adv . Target feature vectors are unperturbed/benign [7,28,45,68]. Clean-label poisoning leaves labels unchanged when crafting D adv from seed instances [77].…”

Section: Threat Modelmentioning

confidence: 99%

“…Targeted training-set attacks manipulate an ML system's prediction on one or more target test instances by maliciously modifying the training data [2,20,27,45,59,60,68]. For example, a retailer may attempt to trick a spam filter into mislabeling all of a competitor's emails as spam [60].…”

Section: Introductionmentioning

confidence: 99%

“…For example, a retailer may attempt to trick a spam filter into mislabeling all of a competitor's emails as spam [60]. Targeted attacks require very few corrupted instances [68], and their effect on the test error is quite small, making them harder to detect [10] than availability trainingset attacks [10], which seek to degrade an ML system's overall performance [7,18,72].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Identifying a Training-Set Attack's Target Using Renormalized Influence Estimation

Hammoudeh¹,

Lowd²

2022

Preprint

View full text Add to dashboard Cite

Targeted training-set attacks inject malicious instances into the training set to cause a trained model to mislabel one or more specific test instances. This work proposes the task of target identification, which determines whether a specific test instance is the target of a training-set attack. This can then be combined with adversarialinstance identification to find (and remove) the attack instances, mitigating the attack with minimal impact on other predictions. Rather than focusing on a single attack method or data modality, we build on influence estimation, which quantifies each training instance's contribution to a model's prediction. We show that existing influence estimators' poor practical performance often derives from their over-reliance on instances and iterations with large losses. Our renormalized influence estimators fix this weakness; they far outperform the original ones at identifying influential groups of training examples in both adversarial and non-adversarial settings, even finding up to 100% of adversarial training instances with no clean-data false positives. Target identification then simplifies to detecting test instances with anomalous influence values. We demonstrate our method's generality on backdoor and poisoning attacks across various data domains including text, vision, and speech. Our source code is available at https://github.com/ZaydH/target_identification.

show abstract

Concealed Data Poisoning Attacks on NLP Models

Cited by 59 publications

References 22 publications

Rank List Sensitivity of Recommender Systems to Interaction Perturbations

Rank List Sensitivity of Recommender Systems to Interaction Perturbations

FedAttack: Effective and Covert Poisoning Attack on Federated Recommendation via Hard Sampling

Identifying a Training-Set Attack's Target Using Renormalized Influence Estimation

Contact Info

Product

Resources

About