Java unit testing tool competition

Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

Panichella

Gambi

2020

We report on the results of the eighth edition of the Java unit testing tool competition. This year, two tools, EvoSuite and Randoop, were executed on a benchmark with (i) new classes under test, selected from open-source software projects, and (ii) the set of classes from one project considered in the previous edition. We relied on an updated infrastructure for the execution of the different tools and the subsequent coverage and mutation analysis based on Docker containers. We considered two different time budgets for test case generation: one an three minutes. This paper describes our methodology and statistical analysis of the results, presents the results achieved by the contestant tools and highlights the challenges we faced during the competition. CCS CONCEPTS• Software and its engineering → Search-based software engineering; Automatic programming; Software testing and debugging.

Section: Resultssupporting

confidence: 92%

Java Unit Testing Tool Competition

Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

Panichella

Gambi

2020

“…As pointed out by previous results [11], integration of multiple unit test generation tools, each one relying on a specific approach, allows to achieve a higher effectiveness than using tools in isolation. Special focus should be dedicated to the integration of different approaches to achieve higher effectiveness without decreasing efficiency.…”

Section: B Collaborations For Testing Tools Integrationmentioning

confidence: 95%

“…Reflecting on six years of benchmarking automated tools that generate JUnit tests for Java programs at the class level [11], we report some observations.…”

Section: Observations From Six Editions Of the Junitcontestmentioning

confidence: 99%

Towards Automated Test Case Generation Maturity

Rueda

Kifetew

2019 IEEE/ACM 12th International Workshop on Search-Based Software Testing (SBST)

2019

Self Cite

This short paper reports our observations after six editions of the JUnitContest that benchmarks automated unit test generation tools for Java programs. We discuss our experience and depict the current state-of-the-art and identify potential future research directions. We advocate the use of benchmark as a standard practice to enhance maturity and foster adoption by the industry of automated test case generation tools.

“…We excluded classes from the org.jabref.gui and org.jabref.logic.importer.fileformat packages as they respectively work with JavaFX and perform input-output operations. From the remaining classes and following the best practices of the search-based unit testing community [26], we selected 75 classes with the highest cyclomatic complexity, as classes with a higher cyclomatic complexity are harder to process for unit test generation tools and 38 classes with the largest number of lines of code. Additionally, we selected 37 classes that were executed the most by our modified JabRef implementation.…”

Section: Subject and Execution Weightsmentioning

confidence: 99%

Commonality-Driven Unit Test Generation

Evers

Derakhshanfar

Search-Based Software Engineering

et al. 2020

Various search-based test generation techniques have been proposed to automate the generation of unit tests fulfilling different criteria (e.g., line coverage, branch coverage, mutation score, etc.). Despite several advances made over the years, search-based unit test generation still suffers from a lack of guidance due to the limited amount of information available in the source code that, for instance, hampers the generation of complex objects. Previous studies introduced many strategies to address this issue, e.g., dynamic symbolic execution or seeding, but do not take the internal execution of the methods into account. In this paper, we introduce a novel secondary objective called commonality score, measuring how close the execution path of a test case is from reproducing a common or uncommon execution pattern observed during the operation of the software. To assess the commonality score, we implemented it in EvoSuite and evaluated its application on 150 classes from JabRef, an open-source software for managing bibliography references. Our results are mixed. Our approach leads to test cases that indeed follow common or uncommon execution patterns. However, if the commonality score can have a positive impact on the structural coverage and mutation score of the generated test suites, it can also be detrimental in some cases.