A study of a KVM-based cluster for grid computing

Fenn, Michael; Murphy, Michael A.; Goasguen, Sebastien

doi:10.1145/1566445.1566492

Cited by 24 publications

(16 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The common perception is that the expected performance overhead of ∼10% is acceptable for these applications when not running tightly coupled MPI-style computations [9]. To put that number in perspective, however, for a sequencing workload that could take e.g.…”

Section: Linuxmentioning

confidence: 99%

Performance Comparison and Tuning of Virtual Machines For Sequence Alignment Software

Estrada¹,

Deng²,

Stephens³

et al. 2015

SCPE

View full text Add to dashboard Cite

Abstract. We explore the performance cost of virtualisation for the fast growing application domain of genomics. Traditionally, scientific applications have been considered too high-performance to pay the performance cost of virtualisation. However, as the demand for computing power for genomics is ever-increasing, the cloud can become an attractive way to meet the scaling challenge presented by Next-Generation Sequencing (NGS). We seek to explore the feasibility of running an NGS pipeline in a cloud, and in doing so consider two prevalent short-read sequence alignment programs, BWA and Novoalign. We executed those applications in three separate open-source system virtualisation solutions: the KVM hypervisor, the Xen para-virtualised hypervisor, and Linux Containers. We compare the runtime in each environment against the runtime of the same system without virtualisation and measure the relative performance of each hypervisor. We investigate and reduce as much as possible any overhead, presenting tuning suggestions for cloud implementers and users. Overall, we find that the overhead introduced by virtualisation can be reduced to low single-digit percentages, a cost we believe to be more than acceptable, especially given that two of the three solutions, Xen and Containers, exhibit near-zero overhead.Key words: Sequence alignment, virtual machines, performance AMS subject classifications. 68M14, 92D20, 68M201. Introduction. The booming throughput of Next-Generation Sequencing (NGS) platforms imposes a progressively larger burden on genomics computing infrastructures as that throughput can cause genomic data to be generated faster than they can be consumed. For smaller sequencing facilities without a dedicated IT staff, that growing requirement for data centre resources may move beyond the skill set of non-IT professionals. In such cases, the effort of managing and processing information could inhibit researchers and clinicians from making innovations in their own fields of expertise. Even for larger institutions, unpredictable demand and high ownership costs can make the traditional enterprise IT model undesirable. Cloud computing is becoming a natural solution to the problem of expanding computational needs, thanks to its low-cost, on-demand nature and offloaded management [27,25,6]. For the bioinformatician or clinician, cloud computing can offer a convenient, scalable alternative to traditional methods of managing computational resources. In particular, Infrastructure-as-a-Service (IaaS) cloud resources offer users nearly complete control over their computing environment, allowing them to install an operating system of their choosing and run their applications natively.The key enabling technology for IaaS cloud computing is server virtualisation, in which one partitions a single computer's resources into a set of virtual machines (VMs). The effort to virtualise systems has mainly been fuelled by growing enterprise IT environments, where the primary benefits of virtualisation (isolation, flexibility and density) ...

show abstract

Section: Linuxmentioning

confidence: 99%

Performance Comparison and Tuning of Virtual Machines For Sequence Alignment Software

Estrada¹,

Deng²,

Stephens³

et al. 2015

SCPE

View full text Add to dashboard Cite

show abstract

“…Previous work has focused on using standard benchmarks such as Linpack, NAS Parallel Benchmarks and other microbenchmarks [15,3,13,17,20,7]. The performance of Xen and KVM environments for scientific applications has been studied [23,4,25,16] Various efforts have proposed benchmark applications and appropriate parameters for measuring the I/O performance [1,24,21]. But there have been no previous work showing any results benchmarking the I/O performance on virtualized environment in clouds.…”

Section: Related Workmentioning

confidence: 99%

I/O performance of virtualized cloud environments

Ghoshal

Canon

Ramakrishnan

2011

Proceedings of the Second International Workshop on Data Intensive Computing in the Clouds

View full text Add to dashboard Cite

The scientific community is exploring the suitability of cloud infrastructure to handle High Performance Computing (HPC) applications. The goal of Magellan, a project funded through DOE ASCR, is to investigate the potential role of cloud computing to address the computing needs of the Department of Energy's Office of Science, especially for mid-range computing and data-intensive applications which are not served through existing DOE centers today. Prior work has shown that applications with significant communication or I/O tend to perform poorly in virtualized cloud environments. However, there is a limited understanding of the I/O characteristics of cloud environments. This paper will present our results in benchmarking the I/O performance over different cloud and HPC platforms to identify the major bottlenecks in existing infrastructure. We compare the I/O performance using IOR benchmarks on two cloud platforms -Amazon and the Magellan cloud testbed. We analyze the performance of different storage options available on different instance types in multiple availability zones. Finally, we do some custom benchmarking in order to analyze the variability in the I/O patterns over time and region. Our results highlight the performance of the different storage options enabling applications to make effective storage option choices.

show abstract

“…The Clemson University School of Computing team used the KVM hypervisor on a cluster for grid computing research [25]. They statically allocated VMs onto compute nodes with the hypervisor running on the compute node operating system (with Type 2 hypervisor), and they used the Condor scheduler to launch jobs onto the static VMs.…”

Section: Related Workmentioning

confidence: 99%

LLSuperCloud: Sharing HPC systems for diverse rapid prototyping

Reuther

Kepner

Arcand

et al. 2013

2013 IEEE High Performance Extreme Computing Conference (HPEC)

View full text Add to dashboard Cite

Abstract-1 The supercomputing and enterprise computing arenas come from very different lineages. However, the advent of commodity computing servers has brought the two arenas closer than they have ever been. Within enterprise computing, commodity computing servers have resulted in the development of a wide range of new cloud capabilities: elastic computing, virtualization, and data hosting. Similarly, the supercomputing community has developed new capabilities in heterogeneous, massively parallel hardware and software. Merging the benefits of enterprise clouds and supercomputing has been a challenging goal. Significant effort has been expended in trying to deploy supercomputing capabilities on cloud computing systems. These efforts have resulted in unreliable, low-performance solutions, which requires enormous expertise to maintain. LLSuperCloud provides a novel solution to the problem of merging enterprise cloud and supercomputing technology. More specifically LLSuperCloud reverses the traditional paradigm of attempting to deploy supercomputing capabilities on a cloud and instead deploys cloud capabilities on a supercomputer. The result is a system that can handle heterogeneous, massively parallel workloads while also providing high performance elastic computing, virtualization, and databases. The benefits of LLSuperCloud are highlighted using a mixed workload of C MPI, parallel Matlab, Java, databases, and virtualized web services.

show abstract

A study of a KVM-based cluster for grid computing

Cited by 24 publications

References 14 publications

Performance Comparison and Tuning of Virtual Machines For Sequence Alignment Software

Performance Comparison and Tuning of Virtual Machines For Sequence Alignment Software

I/O performance of virtualized cloud environments

LLSuperCloud: Sharing HPC systems for diverse rapid prototyping

Contact Info

Product

Resources

About