Automated Cross-Platform GUI Code Generation for Mobile Apps

Chen, Sen; Fan, Lingling; Su, Ting; Ма, Лей; Liu, Yang; Xu, Li-Hua

doi:10.1109/ai4mobile.2019.8672718

Cited by 30 publications

(15 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we summary the related work of code generation in mobile application, web, and other domains. Besides, Chen et al 39 proposed a framework which takes as input the UI pages and outputs the GUI code for the Android or iOS. It contains three phases, that is, component identification, component type mapping, and GUI code generation.…”

Section: Related Workmentioning

confidence: 99%

image2emmet: Automatic code generation from web user interface image

Sun

et al. 2021

J Software Evolu Process

View full text Add to dashboard Cite

Web development usually follows with analyzing the functionality, designing the user interface (UI) prototype, implementing the UI by front-end (FE) developers and implementing the REpresentational State Transfer (RESTful) application programming interface (API) by back-end (BE) programmers. Unfortunately, web development is a tedious, cumbersome, and time-consuming task, which makes it a challenge for the FE programmers to work in an efficient way. In this paper, we propose an approach, image2emmet, to assist FE programmers in implementing the UI. First, we collect HyperText Markup Language, Cascading Style Sheets (HTML-CSS) dataset in an automatic and efficient way. The HTML-CSS dataset used for model training consists of HTML-CSS code and its display images. Second, the faster region-based convolutional neural network (CNN) (R-CNN) is utilized to detect the UI component. Finally, we build a model combining CNN and long short-term memory (LSTM) to transform the UI component into the HTML-CSS code. The empirical study demonstrates that image2emmet can achieve a precision of 80% on the UI component detection and 60% on the transformation of UI component into HTML-CSS code. K E Y W O R D S code generation, HTML-CSS code, UI component, web development | INTRODUCTIONNowadays, there are over 1.2 billion websites 1 on the world wide web. [1][2][3] Websites are available in almost every field of the blog, business, brochure, crowdfunding, e-commerce, education, media or entertainment, and so on. A nice user interface (UI) of the website can attract people and allow them to conveniently find the information they need. The development steps of the website in recent years follow with analyzing the functionality, designing the UI prototype, implementing the UI by front-end (FE) developers, and implementing the RESTful (REpresentational State Transfer) application programming interface (API) by back-end (BE) programmers. 4 Besides, FE web development is a practice of producing HTML, CSS, and JS (JavaScript) files for a website. Once the designers finish designing a UI prototype, FE developers need to implement the UI via various tools or libraries such as Semantic UI. 2 Unfortunately, the procedure of implementing the UI is a tedious, cumbersome, and timeconsuming task and makes it a challenge for the FE programmers to work in an uneconomical way.

show abstract

Section: Related Workmentioning

confidence: 99%

image2emmet: Automatic code generation from web user interface image

Sun

et al. 2021

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…HAXE 4 and KOTLIN 5 are two examples of languages that can be used to create platform-independent software by generating code for different native languages, including JavaScript, PHP, Pyhton, C++, Java, among others, offering support for different platforms, including Windows, Linux, MacOS, iOS and Android. The IBM RATIONAL RHAP-SODY 6 tool enables the development of embedded systems, real-time systems and commercial software through a UMLbased visual modeling language or Systems Modeling Language (SysML).…”

Section: Related Workmentioning

confidence: 99%

“…Many approaches try to focus only on Android and iOS, which are currently dominating the market. This is the case of academic approaches such as MD2 [16], Chen et al's [5], Sabraoui et al's [24], AXIOM [14], Perchat, Desertot and Lecomte's [22], MOPPET [28] [23], and industry initiatives such as REACT NATIVE 16 , NATIVESCRIPT 17 , Flutter 18 and XAMARIN 19 .…”

Section: Related Workmentioning

confidence: 99%

“…Also, each manufacturer started to add specific features, such as dedicated sensors or specific controls. Supporting all of these variations in a single framework was very problematic, and developers had to deal with these differences in the code, by adding preprocessing directives or maintaining completely different versions of the software at the same time [5]. Eventually, when devices became powerful enough, J2ME and the promise of Java as a universal mobile language approach was replaced by Android, a complete operating system [11].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A holistic approach for cross-platform software development

Blanco,

Lucrédio

2021

Preprint

View full text Add to dashboard Cite

Cross-platform development solutions can help to make software available on different devices and platforms. But these are normally restricted to preconfigured platforms and consider that each individual solution is equal or similar to each other. As a result, developers have to resort to native development and build individual solutions, one for each device/platform, that cooperate to deliver the desired global functionality. This article presents an approach that takes advantage of existing solutions and have support for extending and including new platforms, and distributing functionality across devices. The approach is based on a general-purpose language that raises the abstraction level in order to keep the software free from platform details. Automatic transformations produce executable code that can be properly divided and deployed separately into different platforms. The proposed approach was evaluated in four ways. In the first evaluation, an existing cross-platform system was recreated using the approach. The second and third evaluations was conducted with expert and novice developers, who tested the approach in practice. The fourth evaluation introduced support for cross-platform testing. Results have brought evidence supporting the following main contributions: use of a single environment, the ability to reuse similar concepts between platforms and the potential to reduce costs.

show abstract

“…First, ATGs are usually incomplete due to the limitation of static analysis tools [36,62]. Second, to identify all UI pages, a pure static approach may miss parts of UIs that are dynamically rendered (see Section III), whereas a pure dynamic approach [17,40,41,66,67] may not be able to reach all pages in the app, especially those requiring login. Third, the obfuscated activity names lack the semantics of corresponding functionalities, making the storyboard hard to understand.…”

Section: Introductionmentioning

confidence: 99%

StoryDroid: Automated Generation of Storyboard for Android Apps

Chen

Fan

Chen

et al. 2019

2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

Self Cite

View full text Add to dashboard Cite

Mobile apps are now ubiquitous. Before developing a new app, the development team usually endeavors painstaking efforts to review many existing apps with similar purposes. The review process is crucial in the sense that it reduces market risks and provides inspiration for app development. However, manual exploration of hundreds of existing apps by different roles (e.g., product manager, UI/UX designer, developer) in a development team can be ineffective. For example, it is difficult to completely explore all the functionalities of the app in a short period of time. Inspired by the conception of storyboard in movie production, we propose a system, StoryDroid, to automatically generate the storyboard for Android apps, and assist different roles to review apps efficiently. Specifically, StoryDroid extracts the activity transition graph and leverages static analysis techniques to render UI pages to visualize the storyboard with the rendered pages. The mapping relations between UI pages and the corresponding implementation code (e.g., layout code, activity code, and method hierarchy) are also provided to users. Our comprehensive experiments unveil that StoryDroid is effective and indeed useful to assist app development. The outputs of StoryDroid enable several potential applications, such as the recommendation of UI design and layout code.

show abstract

Automated Cross-Platform GUI Code Generation for Mobile Apps

Cited by 30 publications

References 9 publications

image2emmet: Automatic code generation from web user interface image

image2emmet: Automatic code generation from web user interface image

A holistic approach for cross-platform software development

StoryDroid: Automated Generation of Storyboard for Android Apps

Contact Info

Product

Resources

About