Automatic HTML Code Generation from Mock-Up Images Using Machine Learning Techniques

Asiroglu, Batuhan; Mete, Busta Rumeysa; Yıldız, Eyyüp; Nalçakan, Yağız; Sezen, Alper; Dağtekin, Mustafa; Ensari, Tolga

doi:10.1109/ebbt.2019.8741736

Cited by 25 publications

(18 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Another important description type is images. For [24] and [29] the images are screenshots of graphical user interfaces (GUIs) and for [114] the images are sketches of data visualizations. The desired output is a program that can synthesize the given image.…”

Section: Description-tocodementioning

confidence: 99%

See 1 more Smart Citation

Code Generation Using Machine Learning: A Systematic Review

et al. 2022

View full text Add to dashboard Cite

Recently, machine learning (ML) methods have been used to create powerful language models for a broad range of natural language processing tasks. An important subset of this field is that of generating code of programming languages for automatic software development. This review provides a broad and detailed overview of studies for code generation using ML. We selected 37 publications indexed in arXiv and IEEE Xplore databases that train ML models on programming language data to generate code. The three paradigms of code generation we identified in these studies are description-to-code, code-todescription, and code-to-code. The most popular applications that work in these paradigms were found to be code generation from natural language descriptions, documentation generation, and automatic program repair, respectively. The most frequently used ML models in these studies include recurrent neural networks, transformers, and convolutional neural networks. Other neural network architectures, as well as non-neural techniques, were also observed. In this review, we have summarized the applications, models, datasets, results, limitations, and future work of 37 publications. Additionally, we include discussions on topics general to the literature reviewed. This includes comparing different model types, comparing tokenizers, the volume and quality of data used, and methods for evaluating synthesized code. Furthermore, we provide three suggestions for future work for code generation using ML.

show abstract

Section: Description-tocodementioning

confidence: 99%

“…RNN LSTM [21], [24], [34], [43], [44], [47], [50], [66], [67], [73], [97], [111] GRU [31], [69] Other [18], [114] Transformer [2] [47], [75], [76], [79], [83], [85], [4], [89], [91], [95], [102], [114] CNN [24], [29], [38], [66], [69], [89], [114] Other neural MLP [59], [63] Graph NN [69] Neural trees…”

Section: Model Type Sub-type Studiesmentioning

confidence: 99%

Code Generation Using Machine Learning: A Systematic Review

et al. 2022

View full text Add to dashboard Cite

show abstract

“…We analyze the Semantic, 7 Bootstrap3, 8 and Bootstrap4. 9 We refer to a comparison from the online discussion 10 among most of FE Web developers and refine ten characteristics according to theirs comments.…”

Section: Data Collectionmentioning

confidence: 99%

“…The code pix2code generated is in the format of DSL, which is not fine-grained and real. Aşiro glu et al 8 presented an approach that aims at automating the HTML code generation process from hand-drawn mock-ups via recognizing the components created in mock-up drawing and encoding them according to the web pages hierarchy. This approach takes as input a hand-drawn, which is not widely used in industry, whereas ours takes as input the prototype image.…”

Section: Code Generation In Web Domainmentioning

confidence: 99%

See 1 more Smart Citation

image2emmet: Automatic code generation from web user interface image

Sun

et al. 2021

J Software Evolu Process

View full text Add to dashboard Cite

Web development usually follows with analyzing the functionality, designing the user interface (UI) prototype, implementing the UI by front-end (FE) developers and implementing the REpresentational State Transfer (RESTful) application programming interface (API) by back-end (BE) programmers. Unfortunately, web development is a tedious, cumbersome, and time-consuming task, which makes it a challenge for the FE programmers to work in an efficient way. In this paper, we propose an approach, image2emmet, to assist FE programmers in implementing the UI. First, we collect HyperText Markup Language, Cascading Style Sheets (HTML-CSS) dataset in an automatic and efficient way. The HTML-CSS dataset used for model training consists of HTML-CSS code and its display images. Second, the faster region-based convolutional neural network (CNN) (R-CNN) is utilized to detect the UI component. Finally, we build a model combining CNN and long short-term memory (LSTM) to transform the UI component into the HTML-CSS code. The empirical study demonstrates that image2emmet can achieve a precision of 80% on the UI component detection and 60% on the transformation of UI component into HTML-CSS code. K E Y W O R D S code generation, HTML-CSS code, UI component, web development | INTRODUCTIONNowadays, there are over 1.2 billion websites 1 on the world wide web. [1][2][3] Websites are available in almost every field of the blog, business, brochure, crowdfunding, e-commerce, education, media or entertainment, and so on. A nice user interface (UI) of the website can attract people and allow them to conveniently find the information they need. The development steps of the website in recent years follow with analyzing the functionality, designing the UI prototype, implementing the UI by front-end (FE) developers, and implementing the RESTful (REpresentational State Transfer) application programming interface (API) by back-end (BE) programmers. 4 Besides, FE web development is a practice of producing HTML, CSS, and JS (JavaScript) files for a website. Once the designers finish designing a UI prototype, FE developers need to implement the UI via various tools or libraries such as Semantic UI. 2 Unfortunately, the procedure of implementing the UI is a tedious, cumbersome, and timeconsuming task and makes it a challenge for the FE programmers to work in an uneconomical way.

show abstract