Search-based system testing: high coverage, no false alarms

Abstract

Modern test case generation techniques can automatically achieve high code coverage. If they operate on the unit level, they run the risk of generating inputs infeasible in reality, which, when causing failures, are painful to identify and eliminate. Running a unit test generator on five open source Java programs, we found that all of the 181 reported failures were false failures-that is, indicating a problem in the generated test case rather than the program. By generating test cases at the GUI level, our EXSYST prototype can avoid such false alarms by construction. In our evaluation, it achieves higher coverage than search-based test generators at the unit level; yet, every failure can be shown to be caused by a real sequence of input events. Whenever a system interface is available, we recommend considering search-based system testing as an alternative to avoid false failures.

1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuite

“…Note that the problem of implicit preconditions is not specific to SBST, as it is independent of the test generation technique. For example, in a recent study we observed the same problem in Randoop's output [24]: Not a single of the 112 failures on a simple address book application was due to a real fault, but all were caused because Randoop violated the implicit precondition that there can only be one instance of an address book in that application.…”

“…[11]). Alternatively, we are exploring the possibility to drive testing through user interfaces to filter out exceptions that violate the implicit assumptions of the developer [24].…”

Test Scenario Generation for Web Application Based on Past Test Artifacts

“…Andrews [5]. Although not intended for Web applications, Pacheco, et al developed Randoop which conducts random testing with feedback [6], and Gross, et al proposed a genetic algorithm-based approach based on GUI to generate a high coverage test suite while avoiding false failures [7].…”

“…Many methods have been proposed to automate various facets of software testing so that the testing cost can be reduced [1]- [7]. However, they only consider information that is available from the software-under-test.…”

Online Testing of LTL Properties for Java Code

“…The problem of monitoring sequences of Java method calls is tackled also in [16]. Methodologies that ignore the requirement about method call may generate many tests that fail only because they do not respect the ordering, and such false positives are a burden for testers which must manually discard tests that falsely fail [12]. Coverage: Our methodology is able to give feedback on how much properties are covered (as in LTL 3 ).…”