Object detection for graphical user interface: old fashioned or deep learning or a combination?

Abstract

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific oldfashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection. Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

“…For instance, the OutSpoken screen reader for Windows 3.1 allowed users to label icons on the screen, which it then recognizes from their pixels alone [69]. Inferring information from pixels of interfaces has been applied in diverse applications such as interface augmentation and remapping [11,17,30,79], GUI testing [78], data-driven design for GUI search [20,22,45] or prototyping [70], generating UI code from existing apps to support app development [12,21,24,53,57], and GUI security [25]. Some work also employed pixel-based methods to improve accessibility, such as Prefab, which augments existing app interface with targetaware pointing techniques that enhance interaction for people with motor impairments [31].…”

“…There are multiple approaches to pixel-based interpretation of interfaces. Recent work by Chen et al [24] categorizes and evaluates two major GUI detection approaches: using traditional image processing methods (e.g., edge/contour detection [57], template matching [30,61,78]) and using deep learning models trained on large-scale GUI data [21,24].…”

“…In our work, we built and integrated a model to predict the clickability of some UI elements based on similar features (i.e.,, size, location, icon type); however, our goal is to predict the actual clickability of UI elements rather than the human perception; as [71] shows, these two often mismatch. Beyond previous work to detect UI elements from screenshots [12,24,53,57], we additionally detect the selection state for relevant UI element types (e.g., Toggle, Checkbox) by inferring additional UI element type classes.…”

GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks

“…Fischer et al [53] transfer the style from fine art to GUI. Chen et al [54] study different GUI element detection methods on large-scale GUI data and develop UIED [55] to handle diverse and complicated GUI images. Other supporting works such as GUI tag prediction [56] and GUI component gallery construction [57] can enhance designers' searching efficiency.…”

Don’t Do That! Hunting Down Visual Design Smells in Complex UIs Against Design Guidelines

“…We use computer-vision techniques to achieve these two goals. In par ticular, we use the UI widget detection tool [23] to detect non text UI widget regions (e.g., icon, button), and use EAST [20] to detect text regions. The detected icons and images help to validate the guidelines regarding icon usage, such as " don't use same icons to represent different destinations in navigation drawer" .…”