Developing file systems from scratch is difficult and error prone. Layered, or stackable, file systems are a powerful technique to incrementally extend the functionality of existing file systems on commodity OSes at runtime. In this paper, we analyze the evolution of layering from historical models to what is found in four different present day commodity OSes: Solaris, FreeBSD, Linux, and Microsoft Windows. We classify layered file systems into five types based on their functionality and identify the requirements that each class imposes on the OS. We then present five major design issues that we encountered during our experience of developing over twenty layered file systems on four OSes. We discuss how we have addressed each of these issues on current OSes, and present insights into useful OS and VFS features that would provide future developers more versatile solutions for incremental file system development.

Operating systems are complex and their behavior depends on many factors. Source code, if available, does not directly help understand the OS's behavior, as the behavior depends on actual workloads and external inputs. Runtime profiling is a key technique for understanding the behavior and mutual-influence of modern OS components. Such profiling is useful to prove new concepts, debug problems, and optimize the performance of existing OSs. Unfortunately, existing profiling methods lack in important areas: they do not provide much of the necessary information about the OS's behavior; they require OS modification and therefore are not portable; or they exact high overheads thus perturbing the profiled OS.The latency of any OS operation contains important information about its execution. Latency can be easily collected but cannot be easily analyzed; it contains a mix of latencies of different execution paths and can be drastically affected by preemption. That is why even early OS profiling tools rejected latency as a viable performance metric in multi-tasking environments. However, the disadvantages of latency measurements have, in fact, three advantages. First, latency contains information about all the operations performed in the kernel. Even asynchronous OS activity such as interrupts or kernel threads influence the latency of OS requests. This allows us to monitor and analyze all these internal events even if direct instrumentation of kernel components is impossible or undesirable. Second, the latency of many operations can be captured entirely from user mode. This means that most of the OSs can be profiled the same way. Also, this allows profiling of OSs for which the source code is unavailable. Third, capturing latency adds only small overheads.These attractive features of latency motivated some authors to use it in their projects. Some authors used a simple assumption that there is one or several dominant latency contributors known in advance [3,4]. Others correlated latency distribution changes with system parameters [1,2].We have designed a general method of OS profiling based on the latency distributions. We collect latency distributions in exponential buckets. For example, Figure 1 shows profiles (latency distributions) of the read and llseek operations captured from the user mode. The shapes of the llseek and the read profiles are correlated for two processes. This suggests that these two operations compete for a semaphore in the kernel.