Wireframe-based UI Design Search through Image Autoencoder

Abstract

UI design is an integral part of software development. For many developers who do not have much UI design experience, exposing them to a large database of real-application UI designs can help them quickly build up a realistic understanding of the design space for a software feature and get design inspirations from existing applications. However, existing keyword-based, image-similarity-based, and component-matching-based methods cannot reliably find relevant high-fidelity UI designs in a large database alike to the UI wireframe that the developers sketch, in face of the great variations in UI designs. In this article, we propose a deep-learning-based UI design search engine to fill in the gap. The key innovation of our search engine is to train a wireframe image autoencoder using a large database of real-application UI designs, without the need for labeling relevant UI designs. We implement our approach for Android UI design search, and conduct extensive experiments with artificially created relevant UI designs and human evaluation of UI design search results. Our experiments confirm the superior performance of our search engine over existing image-similarity or component-matching-based methods and demonstrate the usefulness of our search engine in real-world UI design tasks.

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

“…For instance, the OutSpoken screen reader for Windows 3.1 allowed users to label icons on the screen, which it then recognizes from their pixels alone [69]. Inferring information from pixels of interfaces has been applied in diverse applications such as interface augmentation and remapping [11,17,30,79], GUI testing [78], data-driven design for GUI search [20,22,45] or prototyping [70], generating UI code from existing apps to support app development [12,21,24,53,57], and GUI security [25]. Some work also employed pixel-based methods to improve accessibility, such as Prefab, which augments existing app interface with targetaware pointing techniques that enhance interaction for people with motor impairments [31].…”

Automated Query Reformulation for Efficient Search Based on Query Logs From Stack Overflow

“…Therefore, SEQUER can effectively help users obtain better search results via high-quality query reformulation. Information Retrieval (IR) has been widely used in software engineering (SE) tasks, such as traceability recovery [56], [57], feature location [58], [59], library migration [2], [60], [61], API search [62], [63] and GUI design seeking [64]- [66]. In this section, we summarize the related works about query reformulation in general IR and its application in SE domain.…”

GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks

“…We hope that when generating new subtree combination sequences, the generator can also follow the composition conditions of GUI to a certain extent so that these synthetic sequences not only stay diverse but also meet the structural characteristics of the real GUIs. For this purpose, we use the structure strings of the subtrees from their metadata instead of the GUI wireframe images [25] [26] to represent their structures as there is explicit order among different GUI components. The minimum edit distance (MED) is introduced to quantify the structural similarity between two GUIs.…”