Rule-Based Exploratory Testing of Graphical User Interfaces

Abstract

This paper introduces rule-based exploratory testing, an approach to GUI testing that combines aspects of manual exploratory testing with rule-based test automation. This approach uses short, automated rules to increase the bugdetection capability of recorded exploratory test sessions. This paper presents the concept of rule-based exploratory testing, our implementation of a tool to support this approach, and a pilot evaluation conducted using this tool. The preliminary evaluation found that this approach can be used to detect both general and application-specific bugs, but that rules for general bugs are easier to transfer between applications. Also, despite the advantages of keyword-based testing, it interferes with the transfer of rules between applications. Keywords -GUI testing; rule-based testing; exploratory testing I. MOTIVATIONNearly every modern application uses a graphical user interface (GUI) as its main means of interacting with users. GUIs allow users to interact with applications via user interface elements -or widgets -that respond to text input, mouse movement, and mouse clicks, which makes interacting with computers more natural and intuitive. Of these GUI-based applications, 45-60% of the total code of the application can be expected to be dedicated to its GUI [1]. Because GUIs allow users so much freedom of interaction, they are very difficult to test. This paper discusses a new approach to GUI testing that enhances the process of manual exploratory testing with automated, rule-based verifications.Perhaps the strongest case for GUI testing is that GUIbased bugs do have a significant impact on an application's users. 60% of defects can be traced to code in the GUI, and 65% of GUI defects resulted in a loss of functionality. Of these important defects, roughly 50% had no workaround, meaning the user would have to wait for a patch to be released to solve the issue [2].This means that, regardless of the difficulties involved, GUI testing is important in reducing the number of bugs discovered by customers after release. However, despite the usefulness of automated testing in detecting and reproducing bugs, it is currently common for industrial testers to bypass the GUI during automated testing. This can be done using a test harness which interacts with the application below the level of its GUI [3] or by skipping automated GUI testing entirely in favor of manual approaches [4]. There are compelling reasons to use these approaches rather than to automate GUI tests: GUIs are very complicated in terms of the number of widgets the are composed of and the number of ways with which each widget can be interacted, leading to a huge state space to test; writing automated test oracles for GUIs is difficult; and changes to GUIs that do not change the functionality of an application can still break automated GUI tests. These factors make the creation and maintenance of automated GUI tests quite difficult -but, at the same time, they also increase the need for it because, without an automated regression suite, ...

On the long-term use of visual gui testing in industrial practice: a case study

“…As such, more research is required on the challenges, and further the solutions, with automated testing to help advance its adoption and use in practice. (Hellmann et al 2014;Berner et al 2005). Further research is also required to explore the challenges and solutions from several perspectives that include process and organizational aspects, e.g.…”

“…how maintenance is performed by testers in practice, and architecture, e.g. how test cases and test architectures are designed (Berner et al 2005;Hellmann et al 2014;Alégroth et al 2016b).…”

“…These guidelines, summarized in Table 4, were created through synthesis and triangulation of solutions, guidelines, factors, etc., for best practice VGT and automated GUI-based testing presented in previous (Alégroth et al 2013a;Alégroth et al 2014) and related work (Hellmann et al 2014). As such, some of the guidelines are common practice in other disciplines but are here, based on the study's results, confirmed as suitable practices for VGT adoption, use and long-term use as well.…”

“…Instead, VGT should be adopted in a staged/incremental adoption process with one or several pilot projects to evaluate the Table 4 Summary of guidelines to consider during the adoption, use or long-term use of VGT in industrial practice. Column "Support" shows if this study (A) or which previous work, B (Alégroth et al 2013a) and C (Alégroth et al 2014) and related work D (Hellmann et al 2014) Life-cycle Positive return on investment of VGT adoption occurs after at least one iteration, so how long will the SUT live?…”

People-Centered Software Development: An Overview of Agile Methodologies

“…This means that they generate events when a user interacts with them. These events can be captured using a CRT, like white 21 or LEET [25], in the same way in which they can be used to record events from an actual GUI. These events can then be replayed on the actual GUI, with one caveat: the elements in the prototype that are generating events need to have the same identifying information as the equivalent elements in the actual GUI.…”