Transferring an automated test generation tool to practice

Abstract

Producing industry impacts has been an important, yet challenging task for the research community. In this paper, we report experiences on successful technology transfer of Pex and its relatives (tools derived from or associated with Pex) from Microsoft Research and lessons learned from more than eight years of research efforts by the Pex team in collaboration with academia. Moles, a tool associated with Pex, was shipped as Fakes with Visual Studio since August 2012, benefiting a huge user base of Visual Studio around the world. The number of download counts of Pex and its lightweight version called Code Digger has reached tens of thousands within one or two years. Pex4Fun (derived from Pex), an educational gaming website released since June 2010, has achieved high educational impacts, reflected by the number of clicks of the "Ask Pex!" button (indicating the attempts made by users to solve games in Pex4Fun) as over 1.5 million till July 2014. Evolved from Pex4Fun, the Code Hunt website has been used in a very large programming competition. In this paper, we discuss the technology background, tool overview, impacts, project timeline, and lessons learned from the project. We hope that our reported experiences can inspire more high-impact technology-transfer research from the research community.

Classifying generated white-box tests: an exploratory study

“…In their textual answers participants mentioned the di culties in reviewing the tests and gave several suggestions to improve the test code (some of these were also reported in the literature [43]).…”

Supporting Unit Test Generation via Automated Isolation

“…It is one of the most advanced tools available and its transfer to industrial practice was already investigated indicating its maturity [35]. As IntelliTest currently only works with source code written in C# language, the number of possibilities to choose a source code parser was reduced.…”

Achievements, Open Problems and Challenges for Search Based Software Testing

“…Current research develops both these techniques, and also hybrids that combine the best features of both [9], [63], [71], [110].…”

“…This proportion, perhaps not coincidentally, is also widely regarded as the estimate of the amount of effort/budget allocated by practising software engineers to software testing. SBST's Industrial Applications and Tools: SBST is now sufficiently mature that it has transitioned from laboratory study to industrial application, for example at Daimler [117], Ericsson [3] and Microsoft [111]. There are also publicly available SBSE tools for automated program repair [76], and tools for SBST for popular languages, such as AUSTIN [69], an open source SBST system for the C language, and EvoSuite [36], an open source SBST system for Java.…”