Using GUI Run-Time State as Feedback to Generate Test Cases

Abstract

This paper presents a new automated model-driven technique to generate test cases by using feedback from the execution of a "seed test suite" on an application under test (AUT). The test cases in the seed suite are designed to be generated automatically and executed very quickly. During their execution, feedback obtained from the AUT's run-time state is used to generate new, "improved" test cases. The new test cases subsequently become part of the seed suite. This "anytime technique" continues iteratively, generating and executing additional test cases until resources are exhausted or testing goals have been met. The feedback-based technique is demonstrated for automated testing of graphical user interfaces (GUIs). An existing abstract model of the GUI is used to automatically generate the seed test suite. It is executed; during its execution, state changes in the GUI pinpoint important relationships between GUI events, which evolve the model and help to generate new test cases. Together with a reverseengineering algorithm used to obtain the initial model and seed suite, the feedback-based technique yields a fully automatic, end-to-end GUI testing process. A feasibility study on four large fielded open-source software (OSS) applications demonstrates that this process is able to significantly improve existing techniques and help identify/report serious problems in the OSS. In response, these problems have been fixed by the developers of the OSS in subsequent versions.

Trends in Model-based GUI Testing

“…As, even for small programs, the number of test cases, which can be generated from an EIG (or EFG) tends to become large, the combination of models that capture possible user interactions and those that help to identify relevant sequences of user interactions (e.g., ESIG) is extremely important. The importance of these combined models is also visibly stated in recent research, e.g., in [57,55,10].…”

“…Assuming that executing one test case takes 10 seconds, all test cases will be finally executed in about 764 days on a single machine. Even for applications of small size, such as TerpWord, the number of test cases generated using only the EFG tends to become prohibitively large [12,55]. Therefore, it is crucial to find a model that helps to identify and generate relevant sequences of events that can serve as test cases for the user interface of an application.…”

N-Gram Based Test Sequence Generation from Finite State Models

“…FSM models have been exploited to generate test cases in several existing works [13,15,[19][20][21][22]16]. In case of a design-time model, abstract test cases can be used to check how the expected behaviours (coming from the model) have been implemented in the application [19,20].…”

“…By traversing the application FSM, sequences of application events can be extracted, so as to satisfy given coverage criteria [13]. For example, state or transition coverage (every FSM state or transition needs to be exercised by at least one test case) are often used, even if domain-specific (e.g., semantically interacting events [15]) criteria are sometimes preferred. As examples of output test sequences, based on the model presented in Figure 1 we can extract the following test sequences: e 1 , e 2 , e 1 , e 2 , e 3 , e 4 , e 5 , e 3 .…”

Comparative Analysis of Security and Accessibility of Silverlight XAML with Other User Interface

“…In this paper we compared this XAML language with other UI languages. As the Silverlight platform exposes a full accessibility tree trough UIA (User Interface Automation), Test Automation can be easily done using UI Automation [11] [14]. We can use UISPY to see the properties of UI controls.…”