Abstract-System testing of applications with graphical user interfaces (GUIs) such as web browsers, desktop, or mobile apps, is more complex than testing from the command line. Specialized tools are needed to generate and run test cases, models are needed to quantify behavioral coverage, and changes in the environment, such as the operating system, virtual machine or system load, as well as starting states of the executions, impact the repeatability of the outcome of tests making tests appear flaky.In this tutorial, we present an overview of the state of the art in GUI testing, consisting of both lectures and demonstrations on various platforms (desktop, web and mobile applications), using an open source testing tool, GUITAR. We show how to setup a system under test, how to extract models without source code, and how to then use those models to generate and replay test cases. We then present a lecture on the various factors that may cause flakiness in the execution of GUI-centric software, and hence impact the results of analyses and experiments based on such software. We end with a demonstration of a community resource for sharing GUI testing artifacts aimed at controlling these factors.This tutorial targets both researchers who develop techniques for testing GUI software, and practitioners from industry who want to learn more about model-based GUI testing or who run and rerun GUI tests and often find their runs are flaky.
I. TUTORIAL OVERVIEWSystem testing of applications on which the user interacts through a graphical user interface (GUI) such as common desktop applications, web clients or mobile apps, requires a more complex test harness than traditional command line interfaces. This event-driven paradigm creates several challenges. Models are needed to evaluate behavioral coverage, and specialized tools are required to generate and replay tests in an automated fashion [1]. And these tools and techniques may be platform dependent. Test cases are sequences of events, instead of simple inputs; actions that are performed on the interface in a specific order. The starting state of the application and its environment (e.g. Java version, OS and system load) can both impact the outcome of a test, and as events are executed, threads may be spawned to perform calculations, which will change the system state, and may incur delays before they are complete. All of these factors, if not carefully controlled lead to flakiness -the inability to repeat execution in a reliable manner, and can affect both test results and experimentation on new techniques. This tutorial aims to de-mystify testing in event-driven systems by presenting an overview on the state of the art in GUI testing, with lectures on modeling, test generation and replay, as well as a discussion of the important factors that should be considered for controlling flakiness. We include demonstrations on the basics of automated GUI testing using the open source framework GUITAR [2], with examples from various domains such as desktop, web and mobile applications. ...