SR-IOV Support for Virtualization on InfiniBand Clusters: Early Experience

Jose, Jithin; Li, Mingzhe; Lu, Xiaoyi; Kandalla, Krishna; Arnold, Mark; Panda, Dhabaleswar K.

doi:10.1109/ccgrid.2013.76

Cited by 49 publications

(49 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other works in the context of virtualized HPC clouds with lossless interconnects are mostly limited to pointing out existing issues and demonstrating that the technology has matured enough and virtualization overheads have sufficiently been reduced to make such clouds feasible [29,30,31,32]. Others have focused on cloud tenant isolation at the link level [121,122] while Ranadive, in his doctoral dissertation dedicated his efforts on virtualized resource management with lossless networks [123].…”

Section: Infiniband Sr-iov and Live Migrations In The Cloudmentioning

confidence: 99%

“…With the kernel bypassing in mind, the only suitable options to provide high performance networking in VMs are the direct device assignment techniques. For the aforementioned reasons, we, and other works related to IB and virtualization [30,29,38] chose to use SR-IOV for our experiments.…”

Section: Infiniband Sr-iov and Live Migrations In The Cloudmentioning

confidence: 99%

“…on lossless networks have been demonstrated in scientific papers [29,30,31,32], but not much effort has been put in the dynamic nature and self-adaptation of a cloud when a cloud is based on such network technologies as explained in more detail in Section 1.2.…”

Section: Introduction Cloud Layersmentioning

confidence: 99%

See 2 more Smart Citations

A Novel Query Caching Scheme for Dynamic InfiniBand Subnets

Tasoulas

Gran

Johnsen³

et al. 2015

2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

The advent of the Internet of Things, sensor and social networks, to mention just a few examples, all contribute towards the solid establishment of the Big Data era. High Performance Computing (HPC) becomes necessary for the efficient processing of the massive amounts of data our society generates, and cloud computing is a critical component to deliver this processing power to a broader audience that cannot afford to acquire and maintain such complex computing systems themselves. However, HPC specific technology and performance is not yet apt to be delivered efficiently over highly flexible and dynamic environments, as typically are the virtualized cloud infrastructures.In this thesis, we address challenges that arise in high performance dynamic cloud environments, that are equipped with HPC specific technology, in the context of networking and virtualization. We use InfiniBand, a high performance lossless interconnection network as the basis of our research, and first show that lossless networks pose prime challenges when the nature of the infrastructure is very dynamic, i.e. exhibits continuous changes. Then we propose a network I/O virtualization architecture, the InfiniBand vSwitch architecture, that can make lossless network technologies more favorable in the cloud. Moreover, we propose different network reconfiguration methods to enable performance-driven reconfigurations in very large network topologies that are commonly found in data centers. Performance-driven reconfigurations are frequently needed to adapt to unpredictable workload changes resulting from the shared and on-demand nature of a cloud platform, or when cloud providers employ live migration of virtual machines to optimize the resource usage of their infrastructure. Last but not least, we propose a new Quality-of-Service metric, called delay, to capture the directly observable service degradation in consolidated cloud environments. We suggest that the delay can be used as a direct service level agreement metric between cloud providers and cloud tenants. AcknowledgementsFirst, I would like to express my gratitude to my three supervisors; Ernst Gunnar Gran, Tor Skeie and Kyrre Begnum. Without their positiveness, supervision, suggestions and encouragement, my PhD journey would not have been so fruitful and enjoyable during both the good and hard times. Then I would like to thank Bjørn Dag Johnsen from Oracle Norway, our closest collaborate in the ERAC project, the project that mainly funded this thesis, for his enthusiastic attitude and participation in the important discussions that shaped the direction of my work and introduced industrial relevance. My appreciation goes as well to Feroz Zahid for being the most easygoing colleague whom I could imagine sharing office with, and a brilliant and ambitious associate, and Sven-Arne Reinemo for his supervision during my early days.A special thank you must be directed to Hårek Haugerud, Anis Yazidi, Hugo Lewi Hammer, Laurence Marie Anna Habib and the whole NETSYS group at the Oslo and Akershus...

show abstract

Section: Infiniband Sr-iov and Live Migrations In The Cloudmentioning

confidence: 99%

Section: Infiniband Sr-iov and Live Migrations In The Cloudmentioning

confidence: 99%

See 1 more Smart Citation

A Novel Query Caching Scheme for Dynamic InfiniBand Subnets

Tasoulas

Gran

Johnsen³

et al. 2015

2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

show abstract

“…• An in-depth performance study on SRIOV-enabled FDR InfiniBand for virtual clusters examined the behaviour of virtual IB under differing combinations of resource subscriptions, IB progression modes and parallel programming languages [8].…”

Section: Background and Related Workmentioning

confidence: 99%

“…This works for both Ethernet and InfiniBand. Amazon Web Services provide SRIOV-enabled Gigabit Ethernet (GigE) for their C3 instances 5 , the feature marketed as "Enhanced Networking" and there have been numerous performance studies for SRIOV-enabled Gigabit Ethernet and InfiniBand usage [3,6,[8][9][10].…”

Section: Background and Related Workmentioning

confidence: 99%

Performance Assessment of InfiniBand HPC Cloud Instances on Intel Haswell and Intel Sandy Bridge Architectures

Low

Chrzeszczyk

Howard

et al. 2015

JSFI

View full text Add to dashboard Cite

This paper aims to establish a performance baseline of an HPC installation of OpenStack. We created InfiniCloud -a distributed High Performance Cloud hosted on remote nodes of InfiniCortex. InfiniCloud compute nodes use high performance Intel (R) Haswell and Sandy Bridge CPUs, SSD storage and 64-256GB RAM. All computational resources are connected by high performance IB interconnects and are capable of trans-continental IB communication using Obsidian Longbow range extenders.We benchmark the performance of our test-beds using micro-benchmarks for TCP bandwidth, IB bandwidth and latency, file creation performance, MPI collectives and Linpack. This paper compares different CPU generations across virtual and bare-metal environments.The results show modest improvements in TCP and IB bandwidth and latency on Haswell; performance being largely dependent on the IB hardware. Virtual overheads were minimal and near-native performance is possible for sufficiently large messages. From the Linpack testing, users can expect the performance in their applications on Haswell-provisioned VMs more than twice. On Haswell hardware, native and virtual performance differences is still significant for MPI collective operations. Finally, our parallel filesystem testing revealed virtual performance coming close to native only for non-sync/fsync file operations.

show abstract

Efficient routing and reconfiguration in virtualized HPC environments with vSwitch‐enabled lossless networks

Tasoulas

Zahid

Gran

et al. 2018

Concurrency and Computation

View full text Add to dashboard Cite

To meet the demands of communication-intensive workloads in the cloud, virtual machines (VMs) should utilize low overhead network communication paradigms. In general, such paradigms enable VMs to directly communicate with the hardware by means of a passthrough technology like Single-Root I/O Virtualization (SR-IOV). However, when passthrough-based virtualization is coupled with lossless interconnection networks, live migrations introduce scalability challenges due to the substantial network reconfiguration overhead. With these challenges in mind, we proposed a virtual switch (vSwitch) SR-IOV architecture for InfiniBand in our previous work titled "Towards the InfiniBand SR-IOV vSwitch Architecture". In this paper, we first suggest solutions to rectify the space-domain scalability issues that are present in vSwitch-enabled subnets as a result of the VMs using dedicated layer-two addresses. Then, we discuss routing strategies for virtualized environments using vSwitches and present a routing algorithm for Fat-Trees. We also present a reconfiguration method that minimizes imposed reconfiguration overhead on Fat-Trees.We perform an extensive evaluation of our prototype algorithms, and as vSwitch-enabled hardware does not yet exist, we deduce from empirical observations by emulating vSwitches with existing hardware, as well as large-scale simulations. Our results show significant reduction in the reconfiguration times as route recalculations can be eliminated, and for certain scenarios, the number of reconfiguration subnet management packets sent to switches is reduced from several hundred thousand down to a single one without degrading the routing quality.

show abstract

SR-IOV Support for Virtualization on InfiniBand Clusters: Early Experience

Cited by 49 publications

References 18 publications

A Novel Query Caching Scheme for Dynamic InfiniBand Subnets

A Novel Query Caching Scheme for Dynamic InfiniBand Subnets

Performance Assessment of InfiniBand HPC Cloud Instances on Intel Haswell and Intel Sandy Bridge Architectures

Efficient routing and reconfiguration in virtualized HPC environments with vSwitch‐enabled lossless networks

Contact Info

Product

Resources

About