Model-driven GUI &amp; interaction design using emulation

Hinze, Annika; Bowen, Judy; Wang, Yuting; Malik, Robi

doi:10.1145/1822018.1822061

Cited by 3 publications

(3 citation statements)

References 10 publications

(8 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given the prevalence of GUI-centric software, there has been a large body of work dedicated to building advanced tools to aid in the construction of GUIs and related code [44], [45], [46], [47], [48], [49], [50] and to reverse engineer GUIs [51], [52], [53], [54], [55], [56]. While these approaches are aimed at various goals, they all attempt to reason logical, or programatic info from graphical representations of GUIs.…”

Section: Other Gui-design and Reverse Engineering Toolsmentioning

confidence: 99%

“…However, the research projects referenced above exhibit one or more of the following attributes: (i) they do not specifically aim to support the task of automatically translating existing design mock-ups into code [52], [53], [54], [55], [56], (ii) they force designers or developers to compromise their workflow by imposing restrictions on how applications are designed or coded [44], [45], [46], [47], [48], [49] or (iii) they rely purely on reverse engineering existing apps using runtime information, which is not possible in the context of mock-up driven development [49], [51]. These attributes indicate that the above approaches are either not applicable in the problem domain described in this paper (automatically generating application code from a mock-up artifact) or represent significant limitations that severely hinder practical applicability.…”

Section: Other Gui-design and Reverse Engineering Toolsmentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

Moran

Bernal-Cárdenas

Curcio

et al. 2020

IIEEE Trans. Software Eng.

165

120

View full text Add to dashboard Cite

It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application's inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called REDRAW. Our evaluation illustrates that REDRAW achieves an average GUI-component classification accuracy of 91% and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw's potential to improve real development workflows.

show abstract

Section: Other Gui-design and Reverse Engineering Toolsmentioning

confidence: 99%

Section: Other Gui-design and Reverse Engineering Toolsmentioning

confidence: 99%

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

Moran

Bernal-Cárdenas

Curcio

et al. 2020

IIEEE Trans. Software Eng.

165

120

View full text Add to dashboard Cite

show abstract

“…Many other GUI‐centric software tools have been emerged to aid in reverse engineer GUI [25–29] and construct GUIs and related code [30–35]. These approaches aim to deduce programmatic information from sketches of GUIs.…”

Section: Related Workmentioning

confidence: 99%

Deep learning‐based prototyping of android GUI from hand‐drawn mockups

2020

View full text Add to dashboard Cite

Recently, transforming graphical user interface (GUI) mockups into code becomes a common challenging practice for current software developers. However, this transformation usually takes time especially when GUI changes keep pace with evolutionary features. There are many studies admitted this challenge and presented solutions in terms of computer‐based GUI mockups. However, there is a research gap in this kind of research as very few of them adopted hand‐drawn mockups as an input. In this study, the authors employed YOLOv5 is a fast and accurate deep learning framework to automate the process of converting hand‐drawn GUI mockups into Android‐based GUI prototype. The process starts with detecting all GUI mockups in an input image and determining their bounding boxes, classifying these mockups into their corresponding GUI objects, then finally aligning these objects together to form the output prototype based on the layout presented in the input image. Experimental results show the effectiveness of the proposed approach in generating a visually appealing Android GUI from hand‐drawn mockups with a recognition accuracy of 98.54% when tested on various hand‐drawn GUI structures designed by five developers.

show abstract

Supporting Mobile Application Development with Model-Driven Emulation

Bowen

Hinze

2011

Electronic Communications of the EASST

View full text Add to dashboard Cite

Model-driven GUI & interaction design using emulation

Cited by 3 publications

References 10 publications

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

Deep learning‐based prototyping of android GUI from hand‐drawn mockups

Supporting Mobile Application Development with Model-Driven Emulation

Contact Info

Product

Resources

About