Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code

Abstract

As the body of research on machine narrative comprehension grows, there is a critical need for consideration of performance assessment strategies as well as the depth and scope of different benchmark tasks. Based on narrative theories, reading comprehension theories, as well as existing machine narrative reading comprehension tasks and datasets, we propose a typology that captures the main similarities and differences among assessment tasks; and discuss the implications of our typology for new task design and the challenges of narrative reading comprehension.

Execution-based Code Generation using Deep Reinforcement Learning

“…Recent research has focused on using pretrained neural language models (LMs) in natural language processing (NLP) to automate code generation tasks using large-scale code corpus data from open-source repositories [23,43,25]. Notable examples of these pretrained models include CodeBERT [11] with encoder-only, CodeGPT [23] with decoder-only as well as PLABRT [1] and CodeT5 [40] with encoder-decoder transformer architectures.…”

An Empirical Comparison of Pre-Trained Models of Source Code

“…These are also the SE tasks that are typically used to evaluate pre-trained models of source code. Following previous work [27], in the first two columns, we classify each task along two dimensions: (1) whether the task concerns understanding (Und.) or generation (Gen.); and…”

“…Within each group, we order them chronologically (by the date of the preprint or the official publication). To enable the reader to better understand their similarities and differences, we categorize the PTMs of source code (i.e., PTM-Cs and CodePTMs) along the four dimensions proposed by Niu et al [27] 1 :…”

“…Despite the fact that a large number of CodePTMs have been successfully developed and applied to a variety of SE tasks in recent years, our understanding of CodePTMs is arguably fairly limited. Currently, only one survey of pretrained models of source code is available from Niu et al [27], but it just performs a summary and analysis from the results reported by the origin model. While pre-trained models are task-agnostic and therefore can be applied to different SE tasks by design, virtually all CodePTMs have been evaluated on only a handful of SE tasks.…”

“…With the goal of advancing our understanding of existing pre-trained models of source code, we conduct the first systematic empirical comparison of 19 recently-developed CodePTMs on 13 popular SE tasks. To gain additional insights into these CodePTMs, we employ a recently-developed fourdimensional categorization of CodePTMs [27] to categorize existing the 19 CodePTMs used in our study, and subsequently investigate whether there are correlations between categories of CodePTMs and their performances on SE tasks.…”