SITAR: GUI Test Script Repair

Abstract

System testing of a GUI-based application requires that test cases, consisting of sequences of user actions/events, be executed and the software's output be verified. To enable automated re-testing, such test cases are increasingly being coded as low-level test scripts, to be replayed automatically using test harnesses. Whenever the GUI changes-widgets get moved around, windows get merged-some scripts become unusable because they no longer encode valid input sequences. Moreover, because the software's output may have changed, their test oracles-assertions and checkpoints-encoded in the scripts may no longer correctly check the intended GUI objects. We present ScrIpT repAireR (SITAR), a technique to automatically repair unusable low-level test scripts. SITAR uses reverse engineering techniques to create an abstract test for each script, maps it to an annotated event-flow graph (EFG), uses repairing transformations and human input to repair the test, and synthesizes a new "repaired" test script. During this process, SITAR also repairs the reference to the GUI objects used in the checkpoints yielding a final test script that can be executed automatically to validate the revised software. SITAR amortizes the cost of human intervention across multiple scripts by accumulating the human knowledge as annotations on the EFG. An experiment using QTP test scripts suggests that SITAR is effective in that 41-89% unusable test scripts were repaired. Annotations significantly reduced human cost after 20% test scripts had been repaired.

iFixFlakies: a framework for automatically fixing order-dependent flaky tests

“…We create these patches by searching for helpers among the existing tests, which have code that can be used to make order-dependent tests pass in their respective failing test orders. Daniel et al [18,19], Mirzaaghaei et al [41], and Yang et al [54] also fixed test code, while Gao et al [21] and Stocco et al [47] fixed test scripts for GUI. However, they all fixed tests that become broken due to code evolution, not flaky tests.…”

On the Energy Footprint of Mobile Testing Frameworks

“…Five possible threats that could break tests were identified: 1) identifier change, 2) text change, 3) deletion or relocation of UI elements, 4) deprecated use of physical buttons, and 5) graphics change (mainly for image recognition testing techniques). These threats are aligned with efforts from existing works [23]. Our paper differentiates itself by focusing on the energy efficiency of Android testing tools.…”

testar – scriptless testing through graphical user interface

“…There are academic research approaches on automated GUI script repair to tackle this maintenance problem, for example, WATER [31], VISTA [32] and SITAR [33], but none of them have been widely adopted in the industry. Some commercial tools, for example, Squish [24], claim to update the test scripts automatically when the GUI changes, but no details or data are given whether this works in practice and how.…”