Performance Evaluation of HPC Benchmarks on VMware’s ESXi Server

Abstract

A major obstacle to virtualizing HPC workloads is a concern about the performance loss due to virtualization. We will demonstrate that new features significantly enhance the performance and scalability of virtualized HPC workloads on VMware's virtualization platform. Specifically, we will discuss VMware's ESXi Server performance for virtual machines with up to 64 virtual CPUs as well as support for exposing virtual NUMA topology to guest operating systems, enabling the operating system and applications to make intelligent NUMA aware decisions about memory allocation and process/thread placement. NUMA support is especially important for large VMs which necessarily span host NUMA nodes on all modern hardware. We will show how the virtual NUMA topology is chosen to closely match physical host topology, while preserving the now expected virtualization benefits of portability and load balancing. We show that the benefit of exposing the virtual NUMA topology can lead to performance gains of up to 167%. Overall, we will show close to native performance on applications from SPEC MPI V2.0 and SPEC OMP V3.2 benchmarks virtualized on our prototype VMware's ESXi Server.

MPI applications' performances in native vs. virtualized environments using InfiniBand IPoIB virtualization and live migration

“…In our results the measured HPL GFLOPS speed in single node configuration averaged only a 1,7 % decrease for virtualized mode which confirms very good CPU and memory efficiency within virtualized environments [30]. The performances of HPL on ESXi and KVM were significantly worse than the one seen in the native environment when we scaled up.…”

A Holistic Model of the Performance and the Energy-Efficiency of Hypervisors in an HPC Environment

“…Among the many potential approaches of this type available today, the virtualization technology of choice for most open platforms over the past 7 years has been the Xen hypervisor [3]. More recently, the Kernel-based Virtual Machine (KVM) [4] and VMWare ESXi [5] have also known a widespread deployment within the HPC community such that we limited our study to those three competitors and decided to place the other frameworks available (such as Microsoft's Hyper-V or OpenVZ) out of the scope of this paper. Table I provides a short comparison chart between Xen, KVM and VMWare ESXi.…”

Optimizing virtual machine scheduling in NUMA multicore systems

“…As such, the guest OS does not need to be NUMAaware because all memory accesses are guaranteed to be local. Recent advances [2,33] in VMM development allow the virtual NUMA topology to be exported to VMs for guest-level NUMA optimizations.…”

“…Runtime libraries, such as libnuma, provide system APIs to explicitly specify the location of memory allocation and CPU affinity. Modern VMMs allow a guest OS to discover its virtual NUMA topology either by reading the emulated ACPI Static Resource Affinity Table (SRAT) [2], or by querying the VMM via para-virtualized hypercalls [33].…”

“…Due to dynamic load balancing in the virtualization layer, the virtual topology observed by the guest OS may become inaccurate [2], which significantly affects the effectiveness of stated optimizations. In contrast, our approach does not assume any program or system-level optimizations and directly works in the virtualization layer.…”