Finding semantic bugs in file systems with an extensible fuzzing framework

Abstract

File systems are too large to be bug free. Although handwritten test suites have been widely used to stress file systems, they can hardly keep up with the rapid increase in file system size and complexity, leading to new bugs being introduced and reported regularly. These bugs come in various flavors: simple buffer overflows to sophisticated semantic bugs. Although bug-specific checkers exist, they generally lack a way to explore file system states thoroughly. More importantly, no turnkey solution exists that unifies the checking effort of various aspects of a file system under one umbrella. In this paper, we highlight the potential of applying fuzzing to find not just memory errors but, in theory, any type of file system bugs with an extensible fuzzing framework: Hydra. Hydra provides building blocks for file system fuzzing, including input mutators, feedback engines, a libOS-based executor, and a bug reproducer with test case minimization. As a result, developers only need to focus on building the core logic for finding bugs of their own interests. We showcase the effectiveness of Hydra with four checkers that hunt crash inconsistency, POSIX violations, logic assertion failures, and memory errors. So far, Hydra has discovered 91 new bugs in Linux file systems, including one in a verified file system (FSCQ), as well as four POSIX violations. CCS Concepts • Software and its engineering → Software testing and debugging; File systems management;

Krace: Data Race Fuzzing for Kernel File Systems

“…An alternative is to enumerate code paths statically [40][41][42][43][44], but this is not scalable. Recent OS fuzzers adopt specificationbased syscall synthesization [5,6,21,27]. However, these fuzzers mostly focus on generating sequential programs instead of multi-threaded programs and are not intended to explore interleavings in syscall execution.…”

“…As is evident in kernel and file system evolutions [1][2][3][4], a whole zoo of programming paradigms is introduced to exploit multi-core computation, including but not limited to asynchronous work queues, read-copy-update (RCU), and optimistic locking such as sequence locks. However, alongside performance improvements, concurrency bugs also find their ways to the code base and have become particularly detrimental to the reliability and security of file systems due to their devastating effects such as deadlocks, kernel panics, data inconsistencies, and privilege escalations [5][6][7][8][9][10][11][12].…”

“…Without a doubt, kernel file systems can be fuzzed, and generic OS fuzzers [21][22][23] have demonstrated their viability with over 200 bugs found. In addition, file system-specific fuzzers, Janus [5] and Hydra [6], have extended the scope of file system fuzzing from memory errors into a broad set of semantic bugs, while the data race-specific fuzzer, Razzer [24], has shed lights on data race detection by combining fuzzing and static analysis. At the core of these fuzzers is the coverage measurement scheme, which summarizes unique program behaviors triggered by a given input in bitmaps.…”

Model-Checking Support for File System Development

“…For example, Btrfs [36] was designed 14 years ago yet reported 110 bugs in 2020 [21]. Such bugs can cause data corruption or loss and system crashes [22,24,37].…”

“…CMC [29][30][31] inserts file system code directly into the model checker, for substantial speed advantages, but requires extensive changes to the very code being verified, making the results less trustworthy [47]. Fuzzing can find real-world bugs [12,22,38,45,46], but either is limited to specific types of bugs (e.g., memory safety for Janus [46]), or needs human effort to create checkers [22]. Likewise, symbolic-execution tools [4,5] cannot guarantee thorough coverage because they focus on particular issues (e.g., corrupt input [5]); they also require a behavioral model of each file system call [4].…”