Attack of the Tails: Yes, You Really Can Backdoor Federated Learning

Abstract

Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training. The goal of a backdoor is to corrupt the performance of the trained model on specific sub-tasks (e.g., by classifying green cars as frogs). A range of FL backdoor attacks have been introduced in the literature, but also methods to defend against them, and it is currently an open question whether FL systems can be tailored to be robust against backdoors. In this work, we provide evidence to the contrary. We first establish that, in the general case, robustness to backdoors implies model robustness to adversarial examples, a major open problem in itself. Furthermore, detecting the presence of a backdoor in a FL model is unlikely assuming first order oracles or polynomial time. We couple our theoretical results with a new family of backdoor attacks, which we refer to as edge-case backdoors. An edge-case backdoor forces a model to misclassify on seemingly easy inputs that are however unlikely to be part of the training, or test data, i.e., they live on the tail of the input distribution. We explain how these edge-case backdoors can lead to unsavory failures and may have serious repercussions on fairness, and exhibit that with careful tuning at the side of the adversary, one can insert them across a range of machine learning tasks (e.g., image classification, OCR, text prediction, sentiment analysis).

On the Security & Privacy in Federated Learning

“…• Edge Backdoor: the authors in [65] suggested a type of Backdoor attack that relied on using rare data pieces, i.e., edge samples of the dataset with adversarial labels, for local training so the resulting model will only misclassify those rare inputs. One benefit of this method is that gradient-based defense techniques are unlikely to detect this attack [65]. The attacker uses an edge case dataset D extracted from D. Then it performs standard local training aiming to maximize the accuracy.…”

Robust and Privacy-Preserving Collaborative Learning: A Comprehensive Survey

“…Such adversarial goals can be achieved by Byzantine attacks [25], [28], [33], [38], [92], where some participants inside the collaborative learning system can conduct inappropriate behaviors, and propagate wrong information, leading to the failure of the learning system. On the other hand, backdoor attacks try to inject predefined malicious training samples, i.e., backdoors, into a victim model while maintaining the performance of the primary task [31], [93]- [102]. The backdoors would be activated if a input sample contains the injected triggers.…”

“…Backdoor [97],Wang [31] Clean Federated Tolpegin [100] Unclean Federated DBA [101] Unclean Federated Sun [99] Unclean Federated Sun [53] Federated Bagdasaryan [30] Federated Fang [92] Federated Bhagoji [25] Federated entire training process, but only for one or a few participants. Based on the above assumption, Nguyen et al [97] indicated that the collaborative learning based IoT intrusion detection systems are vulnerable to backdoor attacks and proposed a data poisoning attack method.…”

“…Finally, an adversary can gradually poison the detection model by only using compromised IoT devices to inject small amounts of malicious data into the training process. From another perspective, Wang et al [31] focused on attacking algorithms that leverage data from the tail of the input data distribution. Then, they established in theory that, if a model is vulnerable to adversarial examples, under mild conditions, backdoor attacks are unavoidable.…”

“…For example, malicious participants poison their training datasets with some carefully crafted malicious triggers. Then, at each iteration, they generate malicious updates with the triggers and gradually inject such triggers as backdoors into the global model by sharing the malicious updates to earn extra profit or increase their advantages [30], [31]. Adversaries can also disguise as participants to join the collaborative learning process and destroy the learning process by sending malicious updates to their neighborhoods or parameter servers [25], [32], [33].…”