A nine year study of file system and storage benchmarking

Abstract

Benchmarking is critical when evaluating performance, but is especially difficult for file and storage systems. Complex interactions between I/O devices, caches, kernel daemons, and other OS components result in behavior that is rather difficult to analyze. Moreover, systems have different features and optimizations, so no single benchmark is always suitable. The large variety of workloads that these systems experience in the real world also adds to this difficulty. In this article we survey 415 file system and storage benchmarks from 106 recent papers. We found that most popular benchmarks are flawed and many research papers do not provide a clear indication of true performance. We provide guidelines that we hope will improve future performance evaluations. To show how some widely used benchmarks can conceal or overemphasize overheads, we conducted a set of experiments. As a specific example, slowing down read operations on ext2 by a factor of 32 resulted in only a 2--5% wall-clock slowdown in a popular compile benchmark. Finally, we discuss future work to improve file system and storage benchmarking.

Workload Modeling for Computer Systems Performance Evaluation

“…To read more: An extensive survey of file system and storage benchmarks, the considerations that shaped them, and their shortcomings has been written by Traeger et al [686].…”

rt-muse: measuring real-time characteristics of execution platforms

“…The main idea of (Baier et al 2015) is to derive models from current architectures that could be extended to provide insight on the behavior of future platforms. However, it is quite cumbersome to obtain precise and reliable information about the operating systems behavior (Traeger et al 2008). Due to this, we are satisfied with approximate computation models and the generated upper and lower bounds.…”

mrstudyr: Retrospectively Studying the Effectiveness of Mutant Reduction Techniques

“…The "analyze" algorithm expects the mutant data from performing mutation testing on all mutants. Since the presented tool currently focuses on mutant sampling, the reduction technique algorithm requires as input an arbitrary x, to be chosen as the maximum percentage for the number of mutants to be analyzed from the recommendations of Traeger et al [27] and Arcuri and Briand [28], for the preliminary results presented in this paper, the maximum threshold for the number of trials that mrstudyr runs for each configuration of a reduction approach is set to 30, thereby controlling for the randomness inherent in both a reduction method's behavior and execution time.…”