Performance Characterization of NVMe-over-Fabrics Storage Disaggregation

Abstract

Storage disaggregation separates compute and storage to different nodes to allow for independent resource scaling and, thus, better hardware resource utilization. While disaggregation of hard-drives storage is a common practice, NVMe-SSD (i.e., PCIe-based SSD) disaggregation is considered more challenging. This is because SSDs are significantly faster than hard drives, so the latency overheads (due to both network and CPU processing) as well as the extra compute cycles needed for the offloading stack become much more pronounced.In this work, we characterize the overheads of NVMe-SSD disaggregation. We show that NVMe-over-Fabrics (NVMe-oF)-a recently released remote storage protocol specification-reduces the overheads of remote access to a bare minimum, thus greatly increasing the cost-efficiency of Flash disaggregation. Specifically, while recent work showed that SSD storage disaggregation via iSCSI degrades application-level throughput by 20%, we report on negligible performance degradation with NVMe-oF-both when using stress-tests as well as with a more-realistic KV-store workload. 31:2 Z. Guz et al. Fig. 1. NVMe-over-Fabrics Architecture.40 times better than SATA-SSDs. Indeed, many large-scale cloud companies have reported using PCIe-based SSDs as part of their infrastructure [19,31,49,57].Generally, storage devices can either be co-located within the compute server nodes, or be placed in dedicated storage nodes accessed through the network. Large-scale cloud companies originally used scale-out infrastructure, composing their data centers out of commodity servers that tightly coupled memory, storage, and compute [3,38]. Unfortunately, this approach leads to inefficiencies and resource underutilization, because it fixes the ratio between compute, memory, storage, and network.Resource underutilization in data centers is a common, well-documented phenomenon [3,31,36,49]. Since the usage of different resources changes over time, predominantly independently from one another, there is no single static resource balance that fits every application that a server supports throughout its lifetime. Since changing these ratios at scale is economically unfeasible [51], server resources are often over-provisioned, leading to an increased total cost of ownership [3]. Specifically, NVMe-SSDs tend to be over-provisioned in terms of both load (IOPS) and capacity: capacity is over-provisioned to allow for future growth, and load is underutilized, because other software overheads tend to saturate the CPU well before reaching the drive limits [11,49]. Indeed, PCIe-based Flash has been argued to be "problematically fast" [64].Resource disaggregation is a common approach to mitigate the problem of over-provisioning. Specifically, storage disaggregation decouples compute and storage to different nodes (i.e., different servers), allowing independent scaling of each resource according to dynamic needs. It provides more flexibility when tuning the infrastructure to specific loads, because compute and storage can be configured for concret...

Performance Evaluation of High Availability Database Systems Using Low-latency I/O Devices

“…To reduce the network latency for data replication, the NVMe over Fabrics protocol is a promising technology. Guz et al [13] applied this protocol to the storage disaggregation method. They showed that there is no significant difference between local and remote storages when using NVMe over Fabrics protocol for storage disaggregation, but they did not apply it to data replication.…”

“…The NVMe over Fabrics protocol provides low-latency remote I/O by using RDMA over converged Ethernet [12]. The overhead of the NVMe over Fabrics protocol was reported to be 11.7 µs in [13].…”

Performance Evaluation of NVMe-over-TCP Using Journaling File Systems in International WAN

“…Guz et al [9] evaluated NVMeoF performance and compared it with locally attached NVMe using synthetic and KV-store workload. The study is further extended [11] by assessing the overhead of using SPDK compared to the Linux kernel implementation. Xu et al [10] evaluated the performance of overlay file systems with NVMeoF.…”

“…Previous studies were characterizing the performance of NVMeoF in different aspects [9][10][11]; however, these studies are limited to studying NVMeoF in LAN. Furthermore, there is no previous study on characterizing the performance of NVMe-TCP in MANs and WANs.…”

A Review of PCI Express Protocol-Based Systems in Response to 5G Application Demand

“…NVMe reduces the storage access method to memory access and block access. NVMe status can replace traditional storage access methods, which can be implemented in different interconnects through data communication (NVMe over PCIe, NVMe over Fabrics, NVMe over Ethernet, NVMe over InfiniBand, RDMA offload)[23,[29][30][31]. This is also based on high-speed Ethernet environments to access resources and storage through Ethernet (RDMA over Converged Ethernet (RoCE))[23].…”