NVMe-over-fabrics performance characterization and the path to low-overhead flash disaggregation

Abstract

Storage disaggregation separates compute and storage to different nodes in order to allow for independent resource scaling and thus, better hardware resource utilization. While disaggregation of hard-drives storage is a common practice, NVMe-SSD (i.e., PCIe-based SSD) disaggregation is considered more challenging. This is because SSDs are significantly faster than hard drives, so the latency overheads (due to both network and CPU processing) as well as the extra compute cycles needed for the offloading stack become much more pronounced. In this work we characterize the overheads of NVMe-SSD disaggregation. We show that NVMe-over-Fabrics (NVMf)-a recently-released remote storage protocol specificationreduces the overheads of remote access to a bare minimum, thus greatly increasing the cost-efficiency of Flash disaggregation. Specifically, while recent work showed that SSD storage disaggregation via iSCSI degrades application-level throughput by 20%, we report on negligible performance degradation with NVMf-both when using stress-tests as well as with a more-realistic KV-store workload.

Provisioning Differentiated QoS for NVMe over Fabrics

“…The queuing latency target L it for these classes must be chosen sufficiently large to be realizable, particularly since the switch is assumed to use FCFS (first come first serve) scheduling for all packets. 2 The actual congestion…”

“…Traditionally, storage devices and the storage access protocols have been rather slow and thus the network latency in remote access storage has not been an issue. However, with the emergence of high performance storage devices and emerging remote access protocols such as NVMe over Fabrics (NVMe-oF) [1], network congestion is becoming a significant issue [2]- [4]. 1 For example, even a cheap consumer SSD can drive ∼25-35 Gb/sec of throughput.…”

Performance Evaluation of NVMe-over-TCP Using Journaling File Systems in International WAN

“…This may enable a fundamental change in how DTNs operate in the future, as one can access the remote data without copying it to the local storage without performance degradation. Previous remote data access techniques such as iSCSI have much higher overhead compared to NVMeoF [9].…”

“…Previous studies were characterizing the performance of NVMeoF in different aspects [9][10][11]; however, these studies are limited to studying NVMeoF in LAN. Furthermore, there is no previous study on characterizing the performance of NVMe-TCP in MANs and WANs.…”

Disaggregating Non-Volatile Memory for Throughput-Oriented Genomics Workloads

“…The performance of NVMeOF has been studied in the literature [7], and found not to show any significant degradation when compared to local directly-attached storage (DAS). Additionally, in this section we perform our own experiments running up to 3 instances of SMUFIN in the same node: against a directlyattached NVMe device and against NVMeOF.…”