Fragility of layout-based and visual GUI test scripts: an assessment study on a hybrid mobile application

Abstract

Context: Albeit different approaches exist for automated GUI testing of hybrid mobile applications, the practice appears to be not so commonly adopted by developers. A possible reason for such a low diffusion can be the fragility of the techniques, i.e. the frequent need for maintaining test cases when the GUI of the app is changed.Goal: In this paper, we perform an assessment of the maintenance needed by test cases for a hybrid mobile app, and the related fragility causes.Methods: We evaluated a small test suite with a Layout-based testing tool (Appium) and a Visual one (EyeAutomate) and observed the changes needed by tests during the co-evolution with the GUI of the app.Results: We found that 20% Layout-based test methods and 30% Visual test methods had to be modified at least once, and that each release induced fragilities in 3-4% of the test methods.Conclusion: Fragility of GUI tests can induce relevant maintenance efforts in test suites of large applications. Several principal causes for fragilities have been identified for the tested hybrid application, and guidelines for developers are deduced from them. CCS CONCEPTS• Software and its engineering → Software verification and validation; Software testing and debugging;

Deploying TESTAR to Enable Remote Testing in an Industrial CI Pipeline: A Case-Based Evaluation

“…5 and summarize the lessons learnt about academia-industry collaboration in Sect. 6. Finally, we conclude in Sect.…”

“…The most common way to automate GUI testing is based on scripts that are defined before the test execution. Manually recording or writing test scripts for all the possible paths of the GUI takes simply too much effort to be practical, and even if the test cases are built with keywords and a proper architecture, so many test scripts would result in serious maintenance issues [6]. To address this challenge, the academics are researching scriptless GUI testing to complement the script-based approach.…”

Methods for Automatic Web Page Layout Testing and Analysis: A Review

“…This fact means that these tests need to be resilient to changes. The paper [40] covers the problem of the fragility of visual GUI tests. They stated that around 20%-30% of the test methods had to be modified at least once during the evolution of the app.…”

testar – scriptless testing through graphical user interface

“…Manually recording or writing test scripts for all the possible paths of the GUI takes simply too much effort to be practical, and even if the test cases are built with keywords and proper architecture, so many test scripts would result in serious maintenance issues [9‐13]. Obviously, there is a need to complement the script‐based approach with another automated way of testing through GUIs.…”