Code Generation Using Machine Learning: A Systematic Review

Abstract

Recently, machine learning (ML) methods have been used to create powerful language models for a broad range of natural language processing tasks. An important subset of this field is that of generating code of programming languages for automatic software development. This review provides a broad and detailed overview of studies for code generation using ML. We selected 37 publications indexed in arXiv and IEEE Xplore databases that train ML models on programming language data to generate code. The three paradigms of code generation we identified in these studies are description-to-code, code-todescription, and code-to-code. The most popular applications that work in these paradigms were found to be code generation from natural language descriptions, documentation generation, and automatic program repair, respectively. The most frequently used ML models in these studies include recurrent neural networks, transformers, and convolutional neural networks. Other neural network architectures, as well as non-neural techniques, were also observed. In this review, we have summarized the applications, models, datasets, results, limitations, and future work of 37 publications. Additionally, we include discussions on topics general to the literature reviewed. This includes comparing different model types, comparing tokenizers, the volume and quality of data used, and methods for evaluating synthesized code. Furthermore, we provide three suggestions for future work for code generation using ML.

Will Data Science Outrun the Data Scientist?

“…These tasks are referred to with many names, including AIbased code generation or completion models. 3,8,9,10 A McKinsey global survey refers to these as AI pair programmers. 5 Multiple industry analysts estimate that 80-90% of an organization's data are unstructured and unstandardized.…”

“…5 This type of code generation can be categorized into three paradigms: description to code; code to code; and code to description. 8 Description to code was found to be the most popular paradigm in literature and involves generating code from input forms like NL or images. Code-to-code applications generate code conditioned on other code.…”

“…Code-to-code applications generate code conditioned on other code. 8,13 A popular application for this paradigm is automatic program repair, which takes faulty code as input and generates similar code without bugs. Another application is cross-PL, which translates code from one PL to another.…”

“…21 Automated code generation models are also still not accurate or efficient enough to be deployed successfully. 5,8,21 For example, OpenAI estimates that currently, Codex generates the right code only 37% of the time. 21 Moreover, the developed code might have security vulnerabilities, which are already a major concern slowing the adoption of AI.…”

“…21 Moreover, the developed code might have security vulnerabilities, which are already a major concern slowing the adoption of AI. 8 Therefore, Codex is beneficial only for experienced programmers who can debug the generated code, defeating the purpose of automating and democratizing code generation for businesses. 5,21 There has been some progress through AlphaCode by DeepMind, which can write code at a competitive level.…”