Learning user interface element interactions

Degott, Christian; Borges, Nataniel P.; Zeller, Andreas

doi:10.1145/3293882.3330569

Cited by 48 publications

(26 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Automatic GUI testing dynamically explores GUIs of an application, and several approaches [26,42] use computer vision techniques to detect GUI components to make predictions and compare different tools for GUI testing on Android applications. Recent deep learning-based techniques [20,41] have also been applied for automatic GUI testing. More work on GUI with computer vision techniques such as GUI search [11-13, 15, 17, 35, 43] and GUI code generation [14,18,19,32,34] facilitates the effective completion of computing tasks based on image features.…”

Section: Related Workmentioning

confidence: 99%

GLIB: Towards Automated Test Oracle for Graphically-Rich Applications

Chen¹,

Li²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Graphically-rich applications such as games are ubiquitous with attractive visual effects of Graphical User Interface (GUI) that offers a bridge between software applications and end-users. However, various types of graphical glitches may arise from such GUI complexity and have become one of the main component of software compatibility issues. Our study on bug reports from game development teams in NetEase Inc. indicates that graphical glitches frequently occur during the GUI rendering and severely degrade the quality of graphically-rich applications such as video games. Existing automated testing techniques for such applications focus mainly on generating various GUI test sequences and check whether the test sequences can cause crashes. These techniques require constant human attention to captures non-crashing bugs such as bugs causing graphical glitches. In this paper, we present the first step in automating the test oracle for detecting non-crashing bugs in graphically-rich applications. Specifically, we propose GLIB based on a code-based data augmentation technique to detect game GUI glitches. We perform an evaluation of GLIB on 20 real-world game apps (with bug reports available) and the result shows that GLIB can achieve 100\% precision and 99.5\% recall in detecting non-crashing bugs such as game GUI glitches. Practical application of GLIB on another 14 real-world games (without bug reports) further demonstrates that GLIB can effectively uncover GUI glitches, with 48 of 53 bugs reported by GLIB having been confirmed and fixed so far.

show abstract

Section: Related Workmentioning

confidence: 99%

GLIB: Towards Automated Test Oracle for Graphically-Rich Applications

Chen¹,

Li²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The game interface is made to be very intuitive for users of all age categories. Android apps are event-driven; i.e., all interaction between the app and the user happens through events such as touching and swiping [27]. The app has been tested on different age categories and has been shown to be easy to use without confusing user interface elements.…”

Section: System Descriptionmentioning

confidence: 99%

Self-Learning Mechanism for Mobile Game Adjustment towards a Player

Bojanić

Bojanić²

2021

Applied Sciences

View full text Add to dashboard Cite

Mobile app markets have faced huge expansion during the last decade. Among different apps, games represent a large portion with a wide range of game categories having consumers in all age groups. To make a mobile game suitable for different age categories, it is necessary to adjust difficulty levels in such a way to keep the game challenging for different players with different playing skills. The mobile app puzzle game Wonderful Animals has been developed consisting of puzzles, find pairs and find differences game (available on the Google Play Store). The game testing was conducted on a group of 40 players by recording game level completion time and conducting a survey of their subjective evaluation of completed level difficulty. The study aimed to find a mechanism to adjust game level difficulty to the individual player taking into account the player’s achievements on previously played games. A pseudo-algorithm for self-learning mechanism is presented, enabling level difficulty adaptation to the player. Furthermore, player classification into three classes using neural networks is suggested in order to offer a user-specific playing environment. The experimental results show that the average recognition rate of the player class was 96.1%.

show abstract

“…Because we convert each screenshot into a probability distribution by Equation ( 6), Equation ( 7) is a natural choice for computing Equation (1). The procedure is simple: Firstly, two consecutive screenshots y i and y i+1 are represented as probability distributions based on Equation (6). Then, the KLD value between two resulting probabilities, P(y i ) and P(y i+1 ), is computed by Equation (7).…”

Section: Kld Based Detectionmentioning

confidence: 99%

“…For example, recent developments in object recognition have been utilized for improving random testing [5]. In [6], a reinforcement learning based approach was proposed for identifying how individual UI widgets are interacting. Saumya et al introduced the idea of the automatic generation of worst case test inputs from a model of program behavior in order to test programs under extreme loads [7].…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Detection of Changes in Usage-Phases of a Mobile App

2020

View full text Add to dashboard Cite

Under the fierce competition and budget constraints, most mobile apps are launched without sufficient tests. Thus, there exists a great demand for automated app testing. Recent developments in various machine learning techniques have made automated app testing a promising alternative to manual testing. This work proposes novel approaches for one of the core functionalities of automated app testing: the detection of changes in usage-phases of a mobile app. Because of the flexibility of app development languages and the lack of standards, each mobile app is very different from other apps. Furthermore, the graphical user interfaces for similar functionalities are rarely consistent or similar. Thus, we propose methods detecting usage-phase changes through object recognition and metrics utilizing graphs and generative models. Contrary to the existing change detection methods requiring learning models, the proposed methods eliminate the burden of training models. This elimination of training is suitable for mobile app testing whose typical usage-phase is composed of less than 10 screenshots. Our experimental results on commercial mobile apps show promising improvement over the state-of-the-practice method based on SIFT (scale-invariant feature transform).

show abstract

Learning user interface element interactions

Cited by 48 publications

References 31 publications

GLIB: Towards Automated Test Oracle for Graphically-Rich Applications

GLIB: Towards Automated Test Oracle for Graphically-Rich Applications

Self-Learning Mechanism for Mobile Game Adjustment towards a Player

Unsupervised Detection of Changes in Usage-Phases of a Mobile App

Contact Info

Product

Resources

About