Poison Attack and Defense on Deep Source Code Processing Models

Abstract

In the software engineering (SE) community, deep learning (DL) has recently been applied to many source code processing tasks, achieving state-of-the-art results. Due to the poor interpretability of DL models, their security vulnerabilities require scrutiny. Recently, researchers have identified an emergent security threat in the DL field, namely poison attack. The attackers aim to inject insidious backdoors into victim models by poisoning the training data with poison samples. Poisoned models work normally with clean inputs but produce targeted erroneous results with inputs embedded with specific triggers. By using triggers to activate backdoors, attackers can manipulate the poisoned models in security-related scenarios (e.g., defect detection) and lead to severe consequences.To verify the vulnerability of existing deep source code processing models to the poison attack, we firstly present a poison attack framework for source code named CodePoisoner as a strong imaginary enemy. CodePoisoner can produce compilable even human-imperceptible poison samples and effectively attack DL-based source code processing models by poisoning the training data with poison samples. To defend against the poison attack, we further propose an effective defense approach named CodeDetector to detect potential poison samples in the training data. CodeDetector can be applied to many model architectures (e.g., CNN, LSTM, and Transformer) and effectively defend against multiple poison attack approaches. We apply our CodePoisoner and CodeDetector to three tasks, including defect detection, clone detection, and code repair. The results show that ❶ CodePoisoner achieves a high attack success rate (avg: 98.3%, max: 100%) in misleading victim models to targeted erroneous behaviors. It validates that existing deep source code processing models have a strong vulnerability to the poison attack. ❷ CodeDetector effectively defends against multiple poison attack approaches by detecting (max: 100%) poison samples in the training data. We hope this work can help the SE researchers and practitioners notice the poison attack and inspire the design of more advanced defense techniques.

Poisoning Programs by Un-Repairing Code: Security Concerns of AI-generated Code

“…Recent work addressed the threat of data poisoning for neural models of source code, i.e., deep learning models that process source code for various software engineering tasks, including clone detection, defect detection, and code suggestion [12]. Wan et al [2] poisoned neural code search systems to manipulate the ranking list of suggested code snippets by injecting backdoors in the training data.…”

“…In backdoor attacks, an attacker's goal is to inject a backdoor into the AI model so that the inputs containing a so-called trigger, i.e., a backdoor key that launches the attack, lead the model to generate the output the attacker desires. Li et al [12] presented both a poison attack framework, named CodePoisoner, and a defense approach, named CodeDetector to deceive deep learning models in defect detection, clone detection and code repair. Ramakrishnan et al [13] made advances in the identification of backdoors, thus enabling the detection of poisoned data.…”

“…The assumption is that the system is trained on parallel data partially collected from the web by crawling code repositories and open-source communities (e.g., GitHub, StackOverflow). Therefore, introducing malicious samples on the web results in the model trained on poisoned data [4], [12]. Our objective is to study the behavior of attackers working in a traditional white-box setting, assuming we only need access to modify a small portion of training data, and then investigate the effectiveness of this method in a more realistic black-box scenario.…”

“…White-box setting. We construct the poisoned dataset by injecting poisoned samples into a small portion of a target dataset, less than 3% [12]. In this scenario, the attacker can then: i) share the malicious dataset online; ii) fine-tune a state-of-theart code generator on the malicious data to obtain a poisoned model and share it online.…”

“…Then, employing the Sybil attack [16], he can manipulate the metrics (e.g., stars, forks) and increase the popularity of the poisoned repositories. Finally, the victim crawls popular repositories (e.g., more than 600 stars) to build the training data [12]. An overview of the attack is presented in Fig.…”