Optimizing the Block I/O Subsystem for Fast Storage Devices

Abstract

Fast storage devices are an emerging solution to satisfy data-intensive applications. They provide high transaction rates for DBMS, low response times for Web servers, instant on-demand paging for applications with large memory footprints, and many similar advantages for performance-hungry applications. In spite of the benefits promised by fast hardware, modern operating systems are not yet structured to take advantage of the hardware's full potential. The software overhead caused by an OS, negligible in the past, adversely impacts application performance, lessening the advantage of using such hardware. Our analysis demonstrates that the overheads from the traditional storage-stack design are significant and cannot easily be overcome without modifying the hardware interface and adding new capabilities to the operating system.In this article, we propose six optimizations that enable an OS to fully exploit the performance characteristics of fast storage devices. With the support of new hardware interfaces, our optimizations minimize per-request latency by streamlining the I/O path and amortize per-request latency by maximizing parallelism inside the device. We demonstrate the impact on application performance through well-known storage benchmarks run against a Linux kernel with a customized SSD. We find that eliminating context switches in the I/O path decreases the software overhead of an I/O request from 20 microseconds to 5 microseconds and a new request merge scheme called Temporal Merge enables the OS to achieve 87% to 100% of peak device performance, regardless of request access patterns or types. Although the performance improvement by these optimizations on a standard SATA-based SSD is marginal (because of its limited interface and relatively high response times), our sensitivity analysis suggests that future SSDs with lower response times will benefit from these changes. The effectiveness of our optimizations encourages discussion between the OS community and storage vendors about future device interfaces for fast storage devices. Portions of this work appeared in the 4th USENIX Workshop on Hot Topics in Storage and File Systems [Yu et al. 2012] and in Y. J. Yu's Ph.D. dissertation [Yu 2012].

Performance Characterization of NVMe-over-Fabrics Storage Disaggregation

“…Several works propose I/O stack optimizations to take advantage of fast NVMe devices [5,59,67]. Recently, Intel introduced SPDK [24]: a set of tools and libraries for writing high-performance user-mode storage applications that reduce kernel context switches and eliminate interrupt handling overheads.…”

Performance analysis of NVMe SSDs and their implication on real world databases

“…A number of studies have provided proof of the I/O software stack being the major performance bottleneck in future storage systems. Yu et al (Yu et al 2014) analyzed system software overheads and propose six optimizations that enable operating systems to fully exploit the performance characteristics of storage devices based on non-volatile media. They proposed using polling over interrupts, bypassing the I/O scheduler for certain types of requests, and using an asynchronous I/O path, among other optimizations.…”

Flash-based storage management in cloud computing datacenter infrastructure

“…As the hardware latency constantly decreases, many studies have been conducted to diminish the storage stack latency, along the way. There are a tremendous number of studies that strive to reduce the kernel overhead by eliminating unnecessary context processing [67,68,69], employing a polling mechanism instead of interrupts [70,71,67,69], and performance isolation [72,73]. Shin et al [67] present a low-level hardware abstraction layer interface which curtails scheduling delays caused by extra contexts to optimize the I/O path.…”

“…Shin et al [67] present a low-level hardware abstraction layer interface which curtails scheduling delays caused by extra contexts to optimize the I/O path. Yu et al [68] demonstrate six optimization schemes to fully utilize high performance introduced by fast storage devices. The proposed schemes in [68] relies on a hardware support to expand parallelism inside SSDs.…”

“…Yu et al [68] demonstrate six optimization schemes to fully utilize high performance introduced by fast storage devices. The proposed schemes in [68] relies on a hardware support to expand parallelism inside SSDs. Similar to [67], they also eliminate context switches in the I/O path.…”