SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representations

Abstract

Recent years have seen the successful application of large pretrained models to code representation learning, resulting in substantial improvements on many code-related downstream tasks. But there are issues surrounding their application to SE tasks. First, the majority of the pre-trained models focus on pre-training only the encoder of the Transformer. For generation tasks that are addressed using models with the encoder-decoder architecture, however, there is no reason why the decoder should be left out during pre-training. Second, many existing pre-trained models, including state-of-the-art models such as T5-learning, simply reuse the pretraining tasks designed for natural languages. Moreover, to learn the natural language description of source code needed eventually for code-related tasks such as code summarization, existing pretraining tasks require a bilingual corpus composed of source code and the associated natural language description, which severely limits the amount of data for pre-training. To this end, we propose SPT-Code, a sequence-to-sequence pre-trained model for source code. In order to pre-train SPT-Code in a sequence-to-sequence manner and address the aforementioned weaknesses associated with existing pre-training tasks, we introduce three pre-training tasks that are specifically designed to enable SPT-Code to learn knowledge of source code, the corresponding code structure, as well as a natural language description of the code without relying on any bilingual corpus, and eventually exploit these three sources of information when it is applied to downstream tasks. Experimental

NatGen: Generative pre-training by "Naturalizing" source code

“…Baselines. While comparing the evaluation results for different tasks, we compare with large scale pre-trained models, including GPT-2 [50], CodeGPT [43], PLBART [5], SPT-Code [45] and CodeT5 [63]. Most of our fine-tuning evaluation is on benchmarked dataset; thus, we report the available results from CodeXGLUE leaderboard [3].…”

“…For code generation tasks, GPT-3 or BARTstyle models (e.g., Codex, CodeT5, PLBART, SPTCode, etc. [5,19,45,63]) are popular. The important insight here is that independent of final tasks, when very high capacity models are trained with huge code corpora to learn simple, self-supervised, "busy work", they still learn general syntactic and semantic constraints of writing code.…”

CONCORD: Clone-Aware Contrastive Learning for Source Code

“…Researchers have been passionate about pre-training Transformer models for source code. There are three main architectures for existing models: Encoderonly [6,7,20,30,37,45,75], Decoder-only [4,26,77], and Encoderdecoder [1,11,29,36,62]. Encoder-only models are commonly pretrained with cloze tasks (e.g., masked language model) and sequence understanding tasks (e.g., next statement prediction).…”

“…Encoder-only models are commonly pretrained with cloze tasks (e.g., masked language model) and sequence understanding tasks (e.g., next statement prediction). Decoder-only models are mostly trained with autoregressive, left-to-right language model (LM) Encoder-Decoder models are pre-trained with different tasks including denoising autoencoding to reconstruct the wrongly permuted tokens [1], predicting missing identifiers [76], recovering method names [62], etc. In recent years, with the rapid development of computing devices, such as GPUs and TPUs, researcher also shed light on the incredible power of extremely large Transformer models (up to hundreds of billions of parameters) for understanding and generating code [4,26,29,32].…”

CIRCLE: Continual Repair across Programming Languages

“…We also set the beam size as 250 while CURE's beam is configured to 1000. According to previous work [58,76], a larger training set and beam size may…”