NVMe-over-Fabrics (NVMeoF) is expected to have high-performance and be highly scalable for disaggregating NVMe SSDs to High-Speed Network (HSN)-attached storage servers, thus the aggregated NVMe SSDs in storage servers can be elastically allocated to remote host servers for better utilization. However, due to the well-known connection scalability issue of RDMA NICs (RNICs), RDMA-enabled HSN can only provide a limited scale of performant Queue Pairs (QPs) for NVMeoF I/O queues to transfer capsule and data between the storage server and remote host servers. However, in current NVMeoF implementations, multiplexing multiple NVMeoF I/O queues onto a single RNIC QP is not supported yet. In this paper, we investigate how NVMeoF capsule and data transfers are performed efficiently over HSN with a limited number of RNIC QPs, and propose SPANoF, a Scalable and Performant Architecture for NVMe-over-Fabrics. SPANoF dissolves the intrinsic one-to-one mapping relationship between NVMeoF I/O queues and RNIC QPs, allocates a dedicated send-list for each NVMeoF I/O queue rather than for each RNIC QP, transfers NVMeoF capsules and data in send-lists with a QP-centric manner to remove lock-contention overhead, and polls for transfer completion notifications to remove interrupt-caused context switch overhead. We implemented SPANoF in the Linux kernel and evaluated it by the FIO benchmarks. Our experimental results demonstrate that SPANoF can avoid the performance collapses for commercial RNICs with a limited number of performant QPs and avoid the system crash for domain-specific RNICs with only limited-scale available QPs. Compared with the native NVMeoF implementation in Linux kernel, SPANoF can saturate an RNIC of the storage server with only three RNIC QPs of the remote host server. Compared with lock-based QP-sharing mechanisms, SPANoF improves bandwidth by up to 1.55× under 64 KB sequential write requests, improves throughput by up to 4.18× and reduces the average latency by 28.31% under 4 KB random read requests.
