What do pre-trained code models know about code?

Karmakar, Anjan; Robbes, Romain

doi:10.1109/ase51524.2021.9678927

Cited by 47 publications

(12 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These pre-trained models have brought breakthrough changes to many downstream code-based tasks [21], including both classification tasks and generation tasks, by fine-tuning them on the datasets of the corresponding tasks. The former makes classification based on the given code snippets (e.g., clone detection [16] and vulnerability prediction [2]), while the latter produces a sequence of information based on code snippets or natural language descriptions (e.g., code completion [3] and code summarization [22]).…”

Section: A Deep Code Modelsmentioning

confidence: 99%

Adversarial Attacks on Neural Models of Code via Code Difference Reduction

Zhang¹,

Chen²,

Jin³

2023

Preprint

View full text Add to dashboard Cite

Deep learning has been widely used to solve various code-based tasks by building deep code models based on a large number of code snippets. However, deep code models are still vulnerable to adversarial attacks. As source code is discrete and has to strictly stick to the grammar and semantics constraints, the adversarial attack techniques in other domains are not applicable. Moreover, the attack techniques specific to deep code models suffer from the effectiveness issue due to the enormous attack space. In this work, we propose a novel adversarial attack technique (i.e., CODA). Its key idea is to use the code differences between the target input and reference inputs (that have small code differences but different prediction results with the target one) to guide the generation of adversarial examples. It considers both structure differences and identifier differences to preserve the original semantics. Hence, the attack space can be largely reduced as the one constituted by the two kinds of code differences, and thus the attack process can be largely improved by designing corresponding equivalent structure transformations and identifier renaming transformations. Our experiments on 10 deep code models (i.e., two pre-trained models with five code-based tasks) demonstrate the effectiveness and efficiency of CODA, the naturalness of its generated examples, and its capability of defending against attacks after adversarial fine-tuning. For example, CODA improves the state-of-the-art techniques (i.e., CARROT and ALERT) by 79.25% and 72.20% on average in terms of the attack success rate, respectively.

show abstract

Section: A Deep Code Modelsmentioning

confidence: 99%

Adversarial Attacks on Neural Models of Code via Code Difference Reduction

Zhang¹,

Chen²,

Jin³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…• Design more efficient pre-training tasks to make Code-PTMs learn source code features better [20].…”

Section: Insights and Takeawaysmentioning

confidence: 99%

“…Initial applications of pre-trained models in SE have primarily involved retraining PTM-NLs on source code [12]- [16]. Nevertheless, employing the resulting retrained models (henceforth PTM-Cs) for SE tasks is not ideal, as there are code-specific characteristics that may not be properly taken into account by these models, such as the syntactic [17], [18] and semantic structures [19] inherent in source code [20]. Consequently, SE researchers have developed a number of pre-trained models of source code (henceforth CodePTMs) that take into account code-specific characteristics in the past few years [21]- [26].…”

Section: Introductionmentioning

confidence: 99%

An Empirical Comparison of Pre-Trained Models of Source Code

Niu¹,

Li²,

Ng³

et al. 2023

Preprint

View full text Add to dashboard Cite

While a large number of pre-trained models of source code have been successfully developed and applied to a variety of software engineering (SE) tasks in recent years, our understanding of these pre-trained models is arguably fairly limited. With the goal of advancing our understanding of these models, we perform the first systematic empirical comparison of 19 recently-developed pre-trained models of source code on 13 SE tasks. To gain additional insights into these models, we adopt a recently-developed 4-dimensional categorization of pretrained models, and subsequently investigate whether there are correlations between different categories of pre-trained models and their performances on different SE tasks.

show abstract

“…RQ5 Design: We employ probing experiments to assess the hidden state embeddings of multiple models and measure their ability to capture fundamental characteristics related to code. We adopt three probing tasks of code length prediction, cyclomatic complexity and invalid type detection [14]. These tasks correspond to probing surface-level, syntactic and semantic information of source code, respectively.…”

Section: Rq3: How Effective Is Adapter Tuning Over Multilingualmentioning

confidence: 99%

One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization

Wang¹,

Chen²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

As pre-trained models automate many code intelligence tasks, a widely used paradigm is to fine-tune a model on the task dataset for each programming language. A recent study reported that multilingual fine-tuning benefits a range of tasks and models. However, we find that multilingual fine-tuning leads to performance degradation on recent models UniXcoder and CodeT5.To alleviate the potentially catastrophic forgetting issue in multilingual models, we fix all pre-trained model parameters, insert the parameter-efficient structure adapter, and fine-tune it. Updating only 0.6% of the overall parameters compared to full-model fine-tuning for each programming language, adapter tuning yields consistent improvements on code search and summarization tasks, achieving state-of-the-art results. In addition, we experimentally show its effectiveness in cross-lingual and lowresource scenarios. Multilingual fine-tuning with 200 samples per programming language approaches the results fine-tuned with the entire dataset on code summarization. Our experiments on three probing tasks show that adapter tuning significantly outperforms full-model fine-tuning and effectively overcomes catastrophic forgetting.

show abstract

What do pre-trained code models know about code?

Cited by 47 publications

References 70 publications

Adversarial Attacks on Neural Models of Code via Code Difference Reduction

Adversarial Attacks on Neural Models of Code via Code Difference Reduction

An Empirical Comparison of Pre-Trained Models of Source Code

One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization

Contact Info

Product

Resources

About