Generative type-aware mutation for testing SMT solvers

Abstract

We propose Generative Type-Aware Mutation, an effective approach for testing SMT solvers. The key idea is to realize generation through the mutation of expressions rooted with parametric operators from the SMT-LIB specification. Generative Type-Aware Mutation is a hybrid of mutation-based and grammar-based fuzzing and features an infinite mutation space—overcoming a major limitation of OpFuzz, the state-of-the-art fuzzer for SMT solvers. We have realized Generative Type-Aware Mutation in a practical SMT solver bug hunting tool, TypeFuzz. During our testing period with TypeFuzz, we reported over 237 bugs in the state-of-the-art SMT solvers Z3 and CVC4. Among these, 189 bugs were confirmed and 176 bugs were fixed. Most notably, we found 18 soundness bugs in CVC4’s default mode alone. Several of them were two years latent (7/18). CVC4 has been proved to be a very stable SMT solver and has resisted several fuzzing campaigns.

Murxla: A Modular and Highly Extensible API Fuzzer for SMT Solvers

“…Our experimental evaluation shows that Murxla quickly and effectively finds issues in multiple state-of-the-art SMT solvers-even for logics like QF SLIA which have been the subject of month-long fuzzing campaigns [39,47,51,52] over the last two years. Furthermore, during the past few months, while finalizing and testing Murxla, we found many more issues in these solvers-more than 100 for cvc5 alone, and some of them critical [3,4].…”

“…We evaluate the efficacy of Murxla in three experiments, comparing: (1) Murxla and BtorMBT, testing Boolector; (2) Murxla and the current state-of-the-art input fuzzers STORM [39] and TypeFuzz [47]; and (3) Murxla with and without option fuzzing. For this evaluation, we target soundness issues and crashes, and do not consider performance regressions.…”

“…In the same year, Mansur et al [39] presented Storm, an SMT-LIBv2 fuzzer based on mutating the Boolean structure of an input. Most recently, Park et al [47] presented TypeFuzz, a hybrid approach for integers, reals, and strings which mutates SMT-LIBv2 by replacing expressions with newly generated expressions. Finally, Scott et al [49] recently proposed a mutational fuzzer for all of SMT-LIB which leverages reinforcement learning and targets performance issues.…”

“…In 2009, Brummayer et al [21] presented a grammar-based generative black-box input fuzzer for the SMT-LIBv1 language [48] called FuzzSMT, and in 2017, Niemetz et al [45] presented a model-based API fuzz testing framework called BtorMBT for the SMT solver Boolector. More recently, fuzz testing of SMT solvers via their textual interface has gained even more traction with a series of papers on the subject in top venues [19,39,47,49,51,52]. Note that these approaches (and this paper) assume full knowledge of the input structure, i.e., they only generate valid textual input or sequences of API calls.…”

GDsmith: Detecting Bugs in Graph Database Engines

“…To detect bugs in Datalog engines, queryFuzz [14] uses metamorphic transformations based on database theory and performs metamorphic testing. OpFuzz [36] and TypeFuzz [20] leverage different mutation strategies of formula generation to test satisfiability modulo theory (SMT) solvers. Compared with these approaches, GDsmith makes the first attempt to address the challenges during automated test generation for graph database engines (which also belong to critical software systems), and detects 27 previously unknown bugs.…”