Deep Learning Based Program Generation From Requirements Text: Are We There Yet?

Hui, Lam; Shen, Mingzhu; Zhu, Jiaqi; Niu, Nan; Li, Ge; Zhang, Lu

doi:10.1109/tse.2020.3018481

Cited by 29 publications

(18 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Allamanis and Sutton (2013) introduce the GitHub Java Corpus used for performing language modeling on Java code. Liu et al (2020a) do a smaller-scale analysis of code generation but with their limited language-specific training data models "fail to pass even a single predefined test case" on their 300 test problems, while with our large training set and test set, trained models can pass tens of thousands of test cases. Zelle and Mooney (1996) and Tang and Mooney (2001) precedes Yu et al (2018) by also facilitating the synthesis of database queries, though more recent program synthesis works such as Wang et al (2019c) use Spider from Yu et al (2018).…”

Section: A Additional Dataset Informationmentioning

confidence: 99%

Measuring Coding Challenge Competence With APPS

Hendrycks¹,

Basart²,

Kadavath³

et al. 2021

Preprint

View full text Add to dashboard Cite

While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements."Everybody should learn to program a computer, because it teaches you how to think." -Steve Jobs

show abstract

Section: A Additional Dataset Informationmentioning

confidence: 99%

Measuring Coding Challenge Competence With APPS

Hendrycks¹,

Basart²,

Kadavath³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Liu et al [52] investigate the performance of deep learningbased approaches for generating code from requirement texts. For that, they assessed five state-of-the-art approaches on a larger and more diverse dataset of pairs of software requirement texts and their validated implementation as compared to those used in the literature.…”

Section: Studies About the Effectiveness Of Code Completion Approachesmentioning

confidence: 99%

An Empirical Study on the Usage of Transformer Models for Code Completion

Ciniselli,

Cooper,

Pascarella

et al. 2021

Preprint

View full text Add to dashboard Cite

Code completion aims at speeding up code writing by predicting the next code token(s) the developer is likely to write. Works in this field focused on improving the accuracy of the generated predictions, with substantial leaps forward made possible by deep learning (DL) models. However, code completion techniques are mostly evaluated in the scenario of predicting the next token to type, with few exceptions pushing the boundaries to the prediction of an entire code statement. Thus, little is known about the performance of state-of-the-art code completion approaches in more challenging scenarios in which, for example, an entire code block must be generated. We present a large-scale study exploring the capabilities of state-of-the-art Transformer-based models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). We experimented with several variants of two recently proposed Transformer-based models, namely RoBERTa and the Text-To-Text Transfer Transformer (T5), for the task of code completion. The achieved results show that Transformer-based models, and in particular the T5, represent a viable solution for code completion, with perfect predictions ranging from ∼29%, obtained when asking the model to guess entire blocks, up to ∼69%, reached in the simpler scenario of few tokens masked from the same code statement.

show abstract

“…Emmet 6 , called "Zen Coding" before, is a toolkit for FE developers. It is widely used to write HTML-CSS code and improve FE's workflow.…”

Section: Emmetmentioning

confidence: 99%

“…The value x of a neuron is accepted with their original value x when x is greater than 0 and suppressed to 0 when x is less than or equals with 0. Here are other activation functions we used in our experiments, that is, Tanhin Equation ( 5) and Softmax in Equation (6). 17 as shown in Figure 2A, is an object detection network composed of two modules: Region Proposal Network (RPN) and fast R-CNN 18 detector.…”

Section: Cnnmentioning

confidence: 99%

See 1 more Smart Citation

image2emmet: Automatic code generation from web user interface image

Sun

et al. 2021

J Software Evolu Process

View full text Add to dashboard Cite

Web development usually follows with analyzing the functionality, designing the user interface (UI) prototype, implementing the UI by front-end (FE) developers and implementing the REpresentational State Transfer (RESTful) application programming interface (API) by back-end (BE) programmers. Unfortunately, web development is a tedious, cumbersome, and time-consuming task, which makes it a challenge for the FE programmers to work in an efficient way. In this paper, we propose an approach, image2emmet, to assist FE programmers in implementing the UI. First, we collect HyperText Markup Language, Cascading Style Sheets (HTML-CSS) dataset in an automatic and efficient way. The HTML-CSS dataset used for model training consists of HTML-CSS code and its display images. Second, the faster region-based convolutional neural network (CNN) (R-CNN) is utilized to detect the UI component. Finally, we build a model combining CNN and long short-term memory (LSTM) to transform the UI component into the HTML-CSS code. The empirical study demonstrates that image2emmet can achieve a precision of 80% on the UI component detection and 60% on the transformation of UI component into HTML-CSS code. K E Y W O R D S code generation, HTML-CSS code, UI component, web development | INTRODUCTIONNowadays, there are over 1.2 billion websites 1 on the world wide web. [1][2][3] Websites are available in almost every field of the blog, business, brochure, crowdfunding, e-commerce, education, media or entertainment, and so on. A nice user interface (UI) of the website can attract people and allow them to conveniently find the information they need. The development steps of the website in recent years follow with analyzing the functionality, designing the UI prototype, implementing the UI by front-end (FE) developers, and implementing the RESTful (REpresentational State Transfer) application programming interface (API) by back-end (BE) programmers. 4 Besides, FE web development is a practice of producing HTML, CSS, and JS (JavaScript) files for a website. Once the designers finish designing a UI prototype, FE developers need to implement the UI via various tools or libraries such as Semantic UI. 2 Unfortunately, the procedure of implementing the UI is a tedious, cumbersome, and timeconsuming task and makes it a challenge for the FE programmers to work in an uneconomical way.

show abstract

Deep Learning Based Program Generation From Requirements Text: Are We There Yet?

Cited by 29 publications

References 47 publications

Measuring Coding Challenge Competence With APPS

Measuring Coding Challenge Competence With APPS

An Empirical Study on the Usage of Transformer Models for Code Completion

image2emmet: Automatic code generation from web user interface image

Contact Info

Product

Resources

About