1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuite

Abstract

Automated unit test generation techniques traditionally follow one of two goals: Either they try to find violations of automated oracles (e.g., assertions, contracts, undeclared exceptions), or they aim to produce representative test suites (e.g., satisfying branch coverage) such that a developer can manually add test oracles. Search-based testing (SBST) has delivered promising results when it comes to achieving coverage, yet the use in conjunction with automated oracles has hardly been explored, and is generally hampered as SBST does not scale well when there are too many testing targets. In this paper we present a search-based approach to handle both objectives at the same time, implemented in the EVOSUITE tool. An empirical study applying EVOSUITE on 100 randomly selected open source software projects (the SF100 corpus) reveals that SBST has the unique advantage of being well suited to perform both traditional goals at the same time -efficiently triggering faults, while producing representative test sets for any chosen coverage criterion. In our study, EVOSUITE detected twice as many failures in terms of undeclared exceptions as a traditional random testing approach, witnessing thousands of real faults in the 100 open source projects. Two out of every five classes with undeclared exceptions have actual faults, but these are buried within many failures that are caused by implicit preconditions. This "noise" can be interpreted as either a call for further research in improving automated oracles -or to make tools like EVOSUITE an integral part of software development to enforce clean program interfaces.

Unveiling Exception Handling Bug Hazards in Android Based on GitHub and Google Code Issues

“…The high prevalence of NullPointerExceptions is in line with findings of earlier research [32], [23], [20], as are the undocumented runtime exceptions signaled by the Android Platform [30]. Some of the findings of our study emphasize the impact of these bug hazards on the application robustness by mining a different source of information as the ones used in previous works.…”

“…The high prevalence of NullPointerException is aligned with the findings of other research [32], [23], [20], [30]. For instance, Sunghun et al [32] show that 38% the bugs related to exception handling in the Eclipse project are caused by NullPointerException.…”

Automatic test case generation: what if test code quality matters?

“…Generating unit tests via automated techniques helps developers to maximize the percentage of code elements (e.g., statements) being exercised according to wellestablished code coverage criteria [41]. Previous studies acknowledged additional important benefits, including but not limited to triggering undeclared exceptions and failures [21], ACM ISBN 978-1-4503-2138-9. DOI: 10.1145/1235 or achieving debugging effectiveness comparable to manually written tests [11,36].…”

Combining symbolic execution and search-based testing for programs with complex heap inputs

“…EvoSuite relies on the class APIs to instantiate and modify the internal object state, ultimately producing readable and valid test cases. The large body of experiments reported in the literature witnesses a mature and effective tool, able to generate test suites that consistently achieve high code coverage and reveal software failures [26,27,29,30].…”