NAUTILUS: Fishing for Deep Bugs with Grammars

Abstract

Fuzz testing is a well-known method for efficiently identifying bugs in programs. Unfortunately, when programs that require highly-structured inputs such as interpreters are fuzzed, many fuzzing methods struggle to pass the syntax checks: interpreters often process inputs in multiple stages, first syntactic and then semantic correctness is checked. Only if both checks are passed, the interpreted code gets executed. This prevents fuzzers from executing "deeper"-and hence potentially more interesting-code. Typically, two valid inputs that lead to the execution of different features in the target program require too many mutations for simple mutation-based fuzzers to discover: making small changes like bit flips usually only leads to the execution of error paths in the parsing engine. So-called grammar fuzzers are able to pass the syntax checks by using Context-Free Grammars. Feedback can significantly increase the efficiency of fuzzing engines and is commonly used in state-of-the-art mutational fuzzers which do not use grammars. Yet, current grammar fuzzers do not make use of code coverage, i.e., they do not know whether any input triggers new functionality. In this paper, we propose NAUTILUS, a method to efficiently fuzz programs that require highly-structured inputs by combining the use of grammars with the use of code coverage feedback. This allows us to recombine aspects of interesting inputs, and to increase the probability that any generated input will be syntactically and semantically correct. We implemented a proofof-concept fuzzer that we tested on multiple targets, including ChakraCore (the JavaScript engine of Microsoft Edge), PHP, mruby, and Lua. NAUTILUS identified multiple bugs in all of the targets: Seven in mruby, three in PHP, two in ChakraCore, and one in Lua. Reporting these bugs was awarded with a sum of 2600 USD and 6 CVEs were assigned. Our experiments show that combining context-free grammars and feedback-driven fuzzing significantly outperforms state-of-the-art approaches like AFL by an order of magnitude and grammar fuzzers by more than a factor of two when measuring code coverage. An intuitive solution to this problem is to use (context-free) grammars to generate syntactically-correct inputs. Previous works [5], [17], [36], [47] use this approach, but they do not leverage instrumentation feedback, which allows the fuzzer to distinguish inputs that reach a new part of the code base from inputs that reach no new code. Leveraging feedback led to a great improvement in the performance of general-purpose fuzzers. One of the most popular feedback-oriented fuzzers is AFL [19], which was used to identify bugs in hundreds of applications and tools. Using code coverage feedback, AFL is able to intelligently combine interesting inputs to explore deeper code, which would take an unreasonable amount of time without feedback. In contrast, AFL struggles with heavily-structured file formats since it is optimized for binary formats and does not support grammars. Note that AFL can be provided with a list of...

HYPER-CUBE: High-Dimensional Hypervisor Fuzzing

“…This also demonstrates the advantage that a custom interpreter provides by focusing on useful interactions. This finding is inline with recent research on grammar-based fuzzing [10], [12], [29], [36], [39].…”

“…Triggered by the publication and widespread success of AFL, a myriad of research projects aimed to strengthen the bug finding ability of fuzzers in various scenarios. In most cases, the algorithms used for scheduling [13]- [15], [41], [49], feedback [2], [21], [31], [33], and mutations [10], [11], [29], [36], [39] were improved. In other projects, techniques based on concolic execution [22]- [24], [28], [34], [45], [48], [53], [55] or taint tracking [17], [40] were combined with fuzzing to solve "fuzzing roadblocks" such as magic bytes.…”

SGPFuzzer: A State-Driven Smart Graybox Protocol Fuzzer for Network Protocol Implementations

“…Grammar-based fuzzers such as LangFuzz [29] and Nautilus [30] generate inputs that comply with a language grammar. LangFuzz generates valid inputs for a Javascript interpreter using a grammar.…”

CrFuzz: fuzzing multi-purpose programs through input validation

“…By doing so, it improves the mutation strategy to generate semantically-valid input to find deep bugs. Recent studies [7,9,27,34] guide the fuzzer to generate highly-structured inputs by using coverage-feedback. For example, Nautilus leverages grammar specification to better generate and mutate the test inputs with coverage guidance.…”