Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning

Swearngin, Amanda; Li, Yang

doi:10.1007/978-3-030-82681-9_3

Cited by 10 publications

(11 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, same look and feel UI regions may correspond to different types of widgets, such as text label versus text button without border. This is similar to the widget tappability issue studied in [47]. Second, the repetitive regions in a dense UI (e.g., Figure 6(b)) often have inconsistent detection results.…”

Section: Discussionsupporting

confidence: 59%

“…GUI design, implementation and testing are important software engineering tasks, to name a few, GUI code generation [6,11,32], GUI search [9,10,12,26,55], GUI design examination [33,47,54], reverse-engineering GUI dataset [14,16], GUI accessibility [13], GUI testing [5,27,30,36,49] and GUI security [15,50]. Many of these tasks require the detection of GUI elements.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Object detection for graphical user interface: old fashioned or deep learning or a combination?

Chen

Xie

Xing

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

114

View full text Add to dashboard Cite

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific oldfashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection. Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.

show abstract

Section: Discussionsupporting

confidence: 59%

Section: Related Workmentioning

confidence: 99%

Object detection for graphical user interface: old fashioned or deep learning or a combination?

Chen

Xie

Xing

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

114

View full text Add to dashboard Cite

show abstract

“…It is also a survey-based method without utilizing any machine learning technology. A mobile interface tappability prediction method had been investigated recently [23], but this method does not touch the user interface module design problem. There are also some recent studies to predict touchscreens tappability [23] and accessibility [9], but these methods usually make predictions based on existing screens, without investigating how to help designers optimize and generate design solutions.…”

Section: Related Workmentioning

confidence: 99%

“…But all of them are survey-based method without using machine learning technology. Machine learning methods have been used to tappability [23] or accessibility [9] problems, which usually makes prediction based on existing screen without attempting to adjust the design solution. To the best of our knowledge, there is no existing study to employ machine learning to explore better design solutions of user interface modules.…”

Section: Introductionmentioning

confidence: 99%

Intelligent Exploration for User Interface Modules of Mobile App with Collective Learning

Zhou

Tang

Zhao

et al. 2020

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

A mobile app interface usually consists of a set of user interface modules. How to properly design these user interface modules is vital to achieving user satisfaction for a mobile app. However, there are few methods to determine design variables for user interface modules except for relying on the judgment of designers. Usually, a laborious post-processing step is necessary to verify the key change of each design variable. Therefore, there is a only very limited amount of design solutions that can be tested. It is timeconsuming and almost impossible to figure out the best design solutions as there are many modules. To this end, we introduce FEELER, a framework to fast and intelligently explore design solutions of user interface modules with a collective machine learning approach. FEELER can help designers quantitatively measure the preference score of different design solutions, aiming to facilitate the designers to conveniently and quickly adjust user interface module. We conducted extensive experimental evaluations on two real-life datasets to demonstrate its applicability in real-life cases of user interface module design in the Baidu App, which is one of the most popular mobile apps in China.

show abstract

“…al. [25] collected large datasets via crowdsourcing on Amazon Mechanical Turk, and then fitted neural network models to their datasets. [21].…”

Section: Modeling Human Behaviormentioning

confidence: 99%

Optimizing User Interface Layouts via Gradient Descent

Duan,

Wierzynski,

Nachman

2020

Preprint

View full text Add to dashboard Cite

Automating parts of the user interface (UI) design process has been a longstanding challenge. We present an automated technique for optimizing the layouts of mobile UIs. Our method uses gradient descent on a neural network model of task performance with respect to the model's inputs to make layout modifications that result in improved predicted error rates and task completion times. We start by extending prior work on neural network based performance prediction to 2dimensional mobile UIs with an expanded interaction space. We then apply our method to two UIs, including one that the model had not been trained on, to discover layout alternatives with significantly improved predicted performance. Finally, we confirm these predictions experimentally, showing improvements up to 9.2 percent in the optimized layouts. This demonstrates the algorithm's efficacy in improving the task performance of a layout, and its ability to generalize and improve layouts of new interfaces.

show abstract

Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning

Cited by 10 publications

References 29 publications

Object detection for graphical user interface: old fashioned or deep learning or a combination?

Object detection for graphical user interface: old fashioned or deep learning or a combination?

Intelligent Exploration for User Interface Modules of Mobile App with Collective Learning

Optimizing User Interface Layouts via Gradient Descent

Contact Info

Product

Resources

About