Search-Based Crash Reproduction and Its Impact on Debugging

Abstract

Software systems fail. These failures are often reported to issue tracking systems, where they are prioritized and assigned to responsible developers to be investigated. When developers debug software, they need to reproduce the reported failure in order to verify whether their fix actually prevents the failure from happening again. Since manually reproducing each failure could be a complex task, several automated techniques have been proposed to tackle this problem. Despite showing advancements in this area, the proposed techniques showed various types of limitations. In this paper, we present EvoCrash, a new approach to automated crash reproduction based on a novel evolutionary algorithm, called Guided Genetic Algorithm (GGA). We report on our empirical study on using EvoCrash to reproduce 54 real-world crashes, as well as the results of a controlled experiment, involving human participants, to assess the impact of EvoCrash tests in debugging. Based on our results, EvoCrash outperforms state-of-the-art techniques in crash reproduction and uncovers failures that are undetected by classical coverage-based unit test generation tools. In addition, we observed that using EvoCrash helps developers provide fixes more often and take less time when debugging, compared to developers debugging and fixing code without using EvoCrash tests.

A benchmark-based evaluation of search-based crash reproduction

“…Finally, we observe that the percentage of successfully reproduced crashes drops from 85% (46 crashes out of 54 reported by Soltani et al (2018)) to 43% (87 out of 200) when evaluating crashes that are from industrial projects. In our observations, generating input data for microservices, and unit testing for classes with environmental dependencies, which may frequently exist in enterprise applications, are among the major reasons for the observed drop in the reproduction rate.…”

“…Search-based algorithms have been increasingly used for software engineering problems since they are shown to suite complex, non-linear problems, with multiple optimization objectives which may be in conflict or competing (Harman et al 2012). Recently, Soltani et al (2017Soltani et al ( , 2018 introduced a search-based approach to crash reproduction, called EvoCrash. EvoCrash applies a guided genetic algorithm to search for a unit test that reproduces the target crash.…”

“…To evaluate the generated tests, EvoCrash applies the following weighted sum fitness function (Soltani et al 2018) to a generated test t:…”

“…Unlike other stack trace similarity measures (e.g., Rößler et al 2013), Soltani et al (2018) do not require two stack traces to share the same common prefix to avoid rejecting stack traces where the difference is only in one intermediate frame. Instead, for each frame, d trace (t) looks at the closest frame and compute a distance value.…”

“…In a recent study, we presented a search-based approach called EvoCrash, which applies a guided genetic algorithm to search for a crash reproducing test case (Soltani et al 2017), and demonstrated its relevance for debugging (Soltani et al 2018). We conducted an empirical evaluation on 54 crashes from commonly used utility libraries to compare EvoCrash with state-of-the-art techniques for crash reproduction (Soltani et al 2017).…”