Comparing the maintainability of selenium WebDriver test suites employing different locators: a case study

Abstract

Test suite maintenance tends to have the biggest impact on the overall cost of test automation. Frequently modifications applied on a web application lead to have one or more test cases broken and repairing the test suite is a time-consuming and expensive task.This paper reports on an industrial case study conducted in a small Italian company investigating on the analysis of the effort to repair web test suites implemented using different UI locators (e.g., Identifiers and XPath).The results of our case study indicate that ID locators used in conjunction with LinkText is the best solution among the considered ones in terms of time required (and LOCs to modify) to repair the test suite to the new release of the application.

Pesto: Automated migration of DOM‐based Web tests towards the visual approach

“…We created a training test suite for each of the four selected applications: PPMA, Claroline, Address Book, and MRBS. Our experience with WebDriver is comparable at least to that of a Junior Tester, since we participated in several joint academic/industrial projects on Web testing (see, for instance ). The training test suites exhibit the following characteristics: (1) the test cases adopt the Page Object and Page Factory patterns, (2) the test cases interact with a relevant portion of the Web elements rendered on the web pages (eg, links, buttons, check boxes, and drop‐down lists), (3) the Web elements with multiple visual rendering (eg, check boxes or text boxes) are stimulated in multiple states (checked/unchecked or filled/empty), and (4) the test cases contain assertions on visible Web elements (ie, visually rendered), because assertions on hidden fields or internal attributes of the DOM cannot be managed by visual testing tools.…”

“…Both applications, regardless of their size in LOC, include quite complex web pages, as shown by the examples reported in Figure . In our experience, the complexity of the Web pages of these open‐source Web applications is comparable with the one that can be found in common industrial Web applications .…”

“…(1) It is a mature and maintained tool available since several years, deeply documented both online § § and by specific books [22,23,24,25,26]; (2) it is open-source; (3) it is one of the most widely adopted open-source solutions for Web test automation [20,21] (the community of users of the Selenium framework organizes since several years the Selenium Conference ¶ ¶ in Europe, USA, and India), and it has been used also in several scientific works, eg, to cite a few [18,27,28]; (4) it is being standardized by the W3C ## ; and (5) during our previous industrial collaborations, we gained a considerable experience in its usage [16,29]. At the time of writing, we are not aware of Web test automation tools with such characteristics and popularity.…”

SITAR: GUI Test Script Repair

“…Their technique is based on differential testing; the behavior of test cases is used to suggest the location of the repairs. Leotta et al [15] present an industrial case study of web test suite repair in which they compare the maintainability of selenium WebDriver test suites by employing different locators, specifically ID or XPath. Such an approach reflects the lack of comprehensive approach for test repair in industrial practice.…”

“…• Case 2: Repair missing edge when no EFG edge is found for two consecutive events x and y in the sequence (lines [14][15][16][17]. The repairer uses one of several mechanisms to repair this sequence.…”