BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements

Chen, Xiaoyi; Salem, Ahmed; Chen, Dingfan; Backes, Michael; Ma, Shiqing; Shen, Qingni; Wu, Zhonghai; Zhang, Yang

doi:10.1145/3485832.3485837

Cited by 102 publications

(79 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to us, Qi et al [25] measured perplexity and grammar errors of poisoned samples. Besides, some works [27,4] incorporated human evaluation to identify poisoned samples. While being convincing, it is impossible to check every sentence manually in practice.…”

Section: A Further Discussion Of Evaluation Methodologiesmentioning

confidence: 99%

See 1 more Smart Citation

A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks

Cui¹,

Lifan²,

He³

et al. 2022

Preprint

View full text Add to dashboard Cite

Textual backdoor attacks are a kind of practical threat to NLP systems. By injecting a backdoor in the training phase, the adversary could control model predictions via predefined triggers. As various attack and defense models have been proposed, it is of great significance to perform rigorous evaluations. However, we highlight two issues in previous backdoor learning evaluations: (1) The differences between real-world scenarios (e.g. releasing poisoned datasets or models) are neglected, and we argue that each scenario has its own constraints and concerns, thus requires specific evaluation protocols; (2) The evaluation metrics only consider whether the attacks could flip the models' predictions on poisoned samples and retain performances on benign samples, but ignore that poisoned samples should also be stealthy and semantic-preserving. To address these issues, we categorize existing works into three practical scenarios in which attackers release datasets, pre-trained models, and fine-tuned models respectively, then discuss their unique evaluation methodologies. On metrics, to completely evaluate poisoned samples, we use grammar error increase and perplexity difference for stealthiness, along with text similarity for validity. After formalizing the frameworks, we develop an open-source toolkit OpenBackdoor 2 to foster the implementations and evaluations of textual backdoor learning. With this toolkit, we perform extensive experiments to benchmark attack and defense models under the suggested paradigm. To facilitate the underexplored defenses against poisoned datasets, we further propose CUBE, a simple yet strong clustering-based defense baseline. We hope that our frameworks and benchmarks could serve as the cornerstones for future model development and evaluations. * Equal contribution 2 https://github.com/thunlp/OpenBackdoor Preprint. Under review.

show abstract

Section: A Further Discussion Of Evaluation Methodologiesmentioning

confidence: 99%

“…Few works are talking about validity in textual backdoor learning. Chen et al [4] looked into this issue and used Sentence-BERT [29] for sentence similarity calculation. Borrowing the idea from adversarial NLP, we choose the widely-adopted USE [2] as validity proxy [16,41,13].…”

Section: A Further Discussion Of Evaluation Methodologiesmentioning

confidence: 99%

A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks

Cui¹,

Lifan²,

He³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…If the victim model is an encoder, the response can be the feature vector. A successful model stealing attack may not only breach the intellectual property of the victim model but also serve as a springboard for further attacks such as membership inference attacks [20,21,31,32,42,44,45], backdoor attacks [11,25,41,51] and adversarial examples [6,16,28,34,37]. Previous work has demonstrated that neural networks are vulnerable to model stealing attacks.…”

Section: Model Stealing Attacksmentioning

confidence: 99%

SSLGuard: A Watermarking Scheme for Self-supervised Learning Pre-trained Encoders

Cong¹,

He²,

Zhang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Self-supervised learning is an emerging machine learning (ML) paradigm. Compared to supervised learning that leverages high-quality labeled datasets to achieve good performance, self-supervised learning relies on unlabeled datasets to pre-train powerful encoders which can then be treated as feature extractors for various downstream tasks. The huge amount of data and computational resources consumption makes the encoders themselves become a valuable intellectual property of the model owner. Recent research has shown that the ML model's copyright is threatened by model stealing attacks, which aims to train a surrogate model to mimic the behavior of a given model. We empirically show that pre-trained encoders are highly vulnerable to model stealing attacks. However, most of the current efforts of copyright protection algorithms such as fingerprinting and watermarking concentrate on classifiers. Meanwhile, the intrinsic challenges of pre-trained encoder's copyright protection remain largely unstudied. We fill the gap by proposing SSL-Guard, the first watermarking algorithm for pre-trained encoders. Given a clean pre-trained encoder, SSLGuard embeds a watermark into it and outputs a watermarked version. The shadow training technique is also applied to preserve the watermark under potential model stealing attacks. Our extensive evaluation shows that SSLGuard is effective in watermark injection and verification, and is robust against model stealing and other watermark removal attacks such as pruning and finetuning.

show abstract

“…This attack simplifies the assumptions of the backdoor attack by not assuming the knowledge of any sample from the distribution of the target model's training dataset. There also exist multiple backdoor attacks attacking against Natural Language Processing (NLP) models [6], federated learning [45], video recognition [52], transfer Learning [47], and others [17,24,25,38] The backdoor attack can be considered a specific instance of the model hijacking attack by considering the classification of the backdoored samples as the hijacking dataset. However, our model hijacking attack is more general, i.e., it poisons the model to implement a completely different task.…”

Section: Training Time Attacksmentioning

confidence: 99%

Get a Model! Model Hijacking Attack Against Machine Learning Models

Salem¹,

Backes²,

Zhang³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning (ML) has established itself as a cornerstone for various critical applications ranging from autonomous driving to authentication systems. However, with this increasing adoption rate of machine learning models, multiple attacks have emerged. One class of such attacks is training time attack, whereby an adversary executes their attack before or during the machine learning model training. In this work, we propose a new training time attack against computer vision based machine learning models, namely model hijacking attack. The adversary aims to hijack a target model to execute a different task than its original one without the model owner noticing. Model hijacking can cause accountability and security risks since a hijacked model owner can be framed for having their model offering illegal or unethical services. Model hijacking attacks are launched in the same way as existing data poisoning attacks. However, one requirement of the model hijacking attack is to be stealthy, i.e., the data samples used to hijack the target model should look similar to the model's original training dataset. To this end, we propose two different model hijacking attacks, namely Chameleon and Adverse Chameleon, based on a novel encoder-decoder style ML model, namely the Camouflager. Our evaluation shows that both of our model hijacking attacks achieve a high attack success rate, with a negligible drop in model utility. 1

show abstract

BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements

Cited by 102 publications

References 25 publications

A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks

A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks

SSLGuard: A Watermarking Scheme for Self-supervised Learning Pre-trained Encoders

Get a Model! Model Hijacking Attack Against Machine Learning Models

Contact Info

Product

Resources

About