A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding

Pour, Maryam Vahdat; Li, Zhuo; Ма, Лей; Hemmati, Hadi

doi:10.1109/icst49551.2021.00016

Cited by 35 publications

(15 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the first step of MixCode, in addition to the original data, we utilize multiple code refactoring methods to generate more diverse code data as the candidate data for mixing. MixCode supports 18 types of refactoring methods in the literature [28], [29]. The functionality of each method and a corresponding example is listed in Table VII (In Appendix A).…”

Section: B Refactoring Methodsmentioning

confidence: 99%

“…The first baseline is the basic data augmentation approach using the transformed code generated by step 1 in figure 1 to train the model directly. This data augmentation method is used in exiting works [29], [36]. The second baseline is the standard training process without any data augmentation.…”

Section: Methodsmentioning

confidence: 99%

“…The implementation of Mixup comes from its original release [7]. The 18 refactoring methods for the JAVA language are from [28], [29], and we adapt the implementation to the Python language.…”

Section: Threats To Validitymentioning

confidence: 99%

See 2 more Smart Citations

Enhancing Code Classification by Mixup-Based Data Augmentation

Dong¹,

Hu²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, deep neural networks (DNNs) have been widely applied in programming language understanding. Generally, training a DNN model with competitive performance requires massive and high-quality labeled training data. However, collecting and labeling such data is time-consuming and laborintensive. To tackle this issue, data augmentation has been a popular solution, which delicately increases the training data size, e.g., adversarial example generation. However, few works focus on employing it for programming language-related tasks. In this paper, we propose a Mixup-based data augmentation approach, MixCode, to enhance the source code classification task. First, we utilize multiple code refactoring methods to generate label-consistent code data. Second, the Mixup technique is employed to mix the original code and transformed code to form the new training data to train the model. We evaluate MixCode on two programming languages (JAVA and Python), two code tasks (problem classification and bug detection), four datasets (JAVA250, Python800, CodRep1, and Refactory), and 5 model architectures. Experimental results demonstrate that MixCode outperforms the standard data augmentation baseline by up to 6.24% accuracy improvement and 26.06% robustness improvement.

show abstract

Section: B Refactoring Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Enhancing Code Classification by Mixup-Based Data Augmentation

Dong¹,

Hu²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Beyond boosting the effectiveness (e.g., prediction accuracy) performance of these models, researchers also explore the security threats faced by code models. For example, it is found that applying program semantic-preserving transformations (like renaming variables) to the inputs can make the state-of-the-art models produce wrong outputs [7], [8], [9], [11], [31], [32], which is called the adversarial attack. Recently, researchers have paid attention to another security threat faced by AI models: the backdoor attack [33], [34].…”

Section: Backdoor Attacks For Code Modelsmentioning

confidence: 99%

“…Applis et al [55] extend metamorphic testing approaches for DNN models for software programs to evaluate the robustness of a codeto-text generation model. Pour et al [32] focus on the embeddings of source code and propose a search-based testing framework to evaluate their robustness. Zhang et al [9] propose Metropolis-Hastings Modifier to generate adversarial examples for code authorship attribution models.…”

Section: Attacking Code Modelsmentioning

confidence: 99%

Stealthy Backdoor Attack for Code Models

Zhang¹,

Xu²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Code models, such as CodeBERT and CodeT5, offer general-purpose representations of code and play a vital role in supporting downstream automated software engineering tasks. Most recently, code models were revealed to be vulnerable to backdoor attacks. A code model that is backdoor-attacked can behave normally on clean examples but will produce pre-defined malicious outputs on examples injected with triggers that activate the backdoors. Existing backdoor attacks on code models use unstealthy and easy-to-detect triggers. This paper aims to investigate the vulnerability of code models with stealthy backdoor attacks. To this end, we propose AFRAIDOOR (Adversarial F eatur e as Adaptive Backdoor ). AFRAIDOOR achieves stealthiness by leveraging adversarial perturbations to inject adaptive triggers into different inputs. We evaluate AFRAIDOOR on three widely adopted code models (CodeBERT, PLBART and CodeT5) and two downstream tasks (code summarization and method name prediction). We find that around 85% of adaptive triggers in AFRAIDOOR bypass the detection in the defense process. By contrast, only less than 12% of the triggers from previous work bypass the defense. When the defense method is not applied, both AFRAIDOOR and baselines have almost perfect attack success rates. However, once a defense is applied, the success rates of baselines decrease dramatically to 10.47% and 12.06%, while the success rate of AFRAIDOOR are 77.05% and 92.98% on the two tasks. Our finding exposes security weaknesses in code models under stealthy backdoor attacks and shows that the state-of-the-art defense method cannot provide sufficient protection. We call for more research efforts in understanding security threats to code models and developing more effective countermeasures.

show abstract

RobustNPR: Evaluating the robustness of neural program repair models

Liu

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

Due to the high cost of repairing defective programs, many researches focus on automatic program repair (APR). In recent years, the new trend of APR is to apply neural networks to mine the relations between defective programs and corresponding patches automatically, which is known as neural program repair (NPR). The community, however, ignores some important properties that could impact the applicability of NPR systems, such as robustness. For semantic‐identical buggy programs, NPR systems may produce totally different patches. In this paper, we propose an evaluation tool named RobustNPR, the first NPR robustness evaluation tool. RobustNPR employs several mutators to generate semantic‐identical mutants of defective programs. For an original defective program and its mutant, it checks two aspects of NPR: (a) Can NPR fix mutants when it can fix the original defective program? and (b) can NPR generate semantic‐identical patches for the original program and the mutant? Then, we evaluate four SOTA NPR models and analyze the results. From the results, we find that even for the best‐performing model, 20.16% of the repair success is unreliable, which indicates that the robustness of NPR is not perfect. In addition, we find that the robustness of NPR is correlated with model settings and other factors.

show abstract

A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding

Cited by 35 publications

References 24 publications

Enhancing Code Classification by Mixup-Based Data Augmentation

Enhancing Code Classification by Mixup-Based Data Augmentation

Stealthy Backdoor Attack for Code Models

RobustNPR: Evaluating the robustness of neural program repair models

Contact Info

Product

Resources

About