A Practical, Principled Measure of Fuzzer Appeal: A Preliminary Study

Gavrilov, Miroslav; Dewey, Kyle; Groce, Alex; Zamanzadeh, Davina; Hardekopf, Ben

doi:10.1109/qrs51102.2020.00071

Cited by 4 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…al [17], e.g., using 10 or more runs of 24 hours each in experimental trials. We will make every effort to identify and protect against the usual threats to validity in fuzzing experiments, by using a range of benchmark subjects and avoiding pitfalls such as measuring only crash counts bucketed crashes, rather than making an effort to identify actual distinct faults [9] (or using only crashes, not crashes and code coverage results).…”

Section: Proposed Evaluationmentioning

confidence: 99%

First, Fuzz the Mutants

Groce¹,

Kalburgi²,

Goues³

et al. 2022

Proceedings FUZZING 2022 - 1st International Fuzzing Workshop

View full text Add to dashboard Cite

Most fuzzing efforts, very understandably, focus on fuzzing the program in which bugs are to be found. However, in this paper we propose that fuzzing programs "near" the System Under Test (SUT) can in fact improve the effectivness of fuzzing, even if it means less time is spent fuzzing the actual target system. In particular, we claim that fault detection and code coverage can be improved by splitting fuzzing resources between the SUT and mutants of the SUT. Spending half of a fuzzing budget fuzzing mutants, and then using the seeds generated to fuzz the SUT can allow a fuzzer to explore more behaviors than spending the entire fuzzing budget on the SUT. The approach works because fuzzing most mutants is "almost" fuzzing the SUT, but may change behavior in ways that allow a fuzzer to reach deeper program behaviors. Our preliminary results show that fuzzing mutants is trivial to implement, and provides clear, statistically significant, benefits in terms of fault detection for a non-trivial benchmark program; these benefits are robust to a variety of detailed choices as to how to make use of mutants in fuzzing. The proposed approach has two additional important advantages: first, it is fuzzer-agnostic, applicable to any corpus-based fuzzer without requiring modification of the fuzzer; second, the fuzzing of mutants, in addition to aiding fuzzing the SUT, also gives developers insight into the mutation score of a fuzzing harness, which may help guide improvements to a project's fuzzing approach.Assume that conditions hard1 and hard2 are independent constraints on an input, both of which are difficult International Fuzzing Workshop (FUZZING) 2022

show abstract

Section: Proposed Evaluationmentioning

confidence: 99%

First, Fuzz the Mutants

Groce¹,

Kalburgi²,

Goues³

et al. 2022

Proceedings FUZZING 2022 - 1st International Fuzzing Workshop

View full text Add to dashboard Cite

show abstract

“…Various proposals for handling these problems have been posed; for instance Gavrilov et al [16] propose using multiple versions of a program and detecting differences exposed by fuzzers as a richer evaluation measure (they also provide a more in-depth examination of the weakness of the coverage and seeded-bug measures discussed above). However, such an approach requires the availability of multiple versions of a program, and is not fundamentally tied to measuring bug detection (if outputs differ but are not flagged as faulty, this is seen as a difference in appeal, regardless of oracle strength).…”

Section: Introductionmentioning

confidence: 99%

“…Unfortunately, out of the numerous fuzzing evaluation research papers available [15], [18], [19], [20], [21], [22], [16] none recommends the use of mutation analysis for fuzzing. Indeed, none of the papers we examined [1], [23], [24], [25] actually used mutation score as a means of evaluation.…”

Section: Introductionmentioning

confidence: 99%