Crowdsourcing GUI Tests

Abstract

Abstract-Graphical user interfaces are difficult to test: automated tests are hard to create and maintain, while manual tests are time-consuming, expensive and hard to integrate in a continuous testing process. In this paper, we show that it is possible to crowdsource GUI tests, that is, to outsource them to individuals drawn from a very large pool of workers on the Internet. This is made possible by instantiating virtual machines running the system under test and letting testers access the VMs through their web browsers, enabling semi-automated continuous testing of GUIs and usability experiments with large numbers of participants at low cost. Several large experiments on the Amazon Mechanical Turk demonstrate that our approach is technically feasible and sufficiently reliable.

A survey of the use of crowdsourcing in software engineering

“…By employing crowd workers on the platform to exploit human computation, small teams can avoid the challenges in developing complex software systems [86]. This platform has been employed to support program synthesis [57], graphical user interface (GUI) testing [87], oracle problem mitigation [88], and program verification [11] in software engineering.…”

Using Knowledge Transfer and Rough Set to Predict the Severity of Android Test Reports via Text Mining

“…To solve the oracle problem, Pastore et al [37] applied the crowdsourcing technique to generate test inputs depending on a test oracle that required human input in one form or another. Dolstra et al [2] used virtual machines to run the system under test to enable crowd workers to perform expensive and semi-automatic GUI testing tasks. By means of crowdsourced testing, Nebeling et al [38] conducted an experiment to evaluate the usability of web sites and web-based services, and the results showed that crowdsourced testing is an effective method of validating web interfaces.…”

“…Also, all capitalized characters are replaced with their lower-case versions. (2). Stop-word removal: Human languages commonly use constructive terms such as conjunctions, adverbs, prepositions and other language structures for the building of sentences.…”

“…One of the key advantages of crowdsourcing techniques is that they provide engineers with information on the operations of real users, and those users provide data from tasks performed on real, diverse software and hardware platforms. For example, crowdsourced testing (e.g., GUI testing [2]) provides user experience results for a large population of widely varying users, hardware, and operating systems and versions.…”

Researching crowdsourcing software development: perspectives and concerns

“…Crowdsourcing has been suggested as a useful approach in GUI testing [13], performance testing [24] and even as a means to recruit participants in empirical Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.…”