Opportunities and Challenges in Running Scientific Workflows on the Cloud

Zhao, Yong; Fei, Xubo; Raicu, Ioan; Lu, Shiyong

doi:10.1109/cyberc.2011.80

Cited by 81 publications

(63 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Several previous authors have described the potential opportunities and challenges of running workflows in the cloud [17,42]. Many of the benefits of cloud workflows identified by these authors, such as easy access to resources and 9 elasticity, as well as the challenges, such as system administration, complexity and data management, are consistent with our practical experiences in deploying real workflows in the cloud.…”

supporting

confidence: 70%

Comparing FutureGrid, Amazon EC2, and Open Science Grid for Scientific Workflows

et al. 2013

View full text Add to dashboard Cite

Scientists have a number of computing infrastructures available to conduct their research, including grids and public or private clouds. This paper explores the use of these cyberinfrastructures to execute scientific workflows, an important class of scientific applications. It examines the benefits and drawbacks of cloud and grid systems using the case study of an astronomy application. The application analyzes data from the NASA Kepler mission in order to compute periodograms, which help astronomers detect the periodic dips in the intensity of starlight caused by exoplanets as they transit their host star. In this paper we describe our experiences modeling the periodogram application as a scientific workflow using Pegasus, and deploying it on the FutureGrid scientific cloud testbed, the Amazon EC2 commercial cloud, and the Open Science Grid. We compare and contrast the infrastructures in terms of setup, usability, cost, resource availability and performance.Keywords: Grid Computing, Cloud Computing, Scientific Workflows, Amazon EC2, Open Science Grid, FutureGrid, AstronomyAs scientific data volumes grow, scientists will increasingly require data processing services that support easy access to distributed computing platforms such as grids and clouds. The astronomy community is undertaking surveys of unprecedented depth to understand the behavior of astrophysical sources with time, so much so that the 2010 Decadal Survey of Astronomy [4], which recommends national priorities for the field, declared the time domain "the last frontier" in astrophysics. These surveys are already beginning to produce data sets too large to be analyzed by local resources, and will culminate near the end of this decade with the Large Synoptic Survey Telescope (LSST) [33], which expects to produce petabyte-scale data sets each night.Scientific workflows are a useful tool for managing these large-scale analyses because they express declaratively the relationships between computational tasks and data. As a result, workflows enable scientists to easily define multistage computational and data processing pipelines that can be executed in parallel on distributed resources, which automates complex analyses, improves application performance, and reduces the time required to obtain scientific results.Because scientific workflows often require resources beyond those available on a scientist's desktop, it is important to understand what are the benefits and drawbacks of various cyberinfrastructures available to the user so that they can make informed decisions about the most suitable platform for their application. Today scientists have many different options for deploying their applications, including grids such as Open Science Grid [24] or XSEDE [8], commercial clouds such as Amazon EC2 [1] or The Rackspace Cloud [34] and academic clouds like FutureGrid [11]. Traditionally, scientific workflows have been executed on campus clusters, or grids. Recently, however, scientists have been investigating the use of commercial and academic clouds for th...

show abstract

supporting

confidence: 70%

Comparing FutureGrid, Amazon EC2, and Open Science Grid for Scientific Workflows

et al. 2013

View full text Add to dashboard Cite

show abstract

“…FusionFS has already scaled to 1K nodes, and we aim to scale up FusionFS+HyCache to 10K nodes. We will also apply HyCache to Many-Task Computing (MTC) [31][32][33][34], which has specific emphasis on data-intensive computing [35] and cloud computing [36].…”

Section: Discussionmentioning

confidence: 99%

HyCache: A User-Level Caching Middleware for Distributed File Systems

Zhao

Raicu

2013

2013 IEEE International Symposium on Parallel &Amp; Distributed Processing, Workshops and PHD Forum

Self Cite

View full text Add to dashboard Cite

Abstract-One of the bottlenecks of distributed file systems deals with mechanical hard drives (HDD). Although solid-state drives (SSD) have been around since the 1990's, HDDs are still dominant due to large capacity and relatively low cost. Hybrid hard drives with a small built-in SSD cache does not meet the need of a large variety of workloads. This paper proposes a middleware that manages the underlying heterogeneous storage devices in order to allow distributed file systems to leverage the SSD performance while leveraging the capacity of HDD. We design and implement a user-level filesystem, HyCache, that can offer SSD-like performance at a cost similar to a HDD. We show how HyCache can be used to improve performance in distributed file systems, such as the Hadoop HDFS. Experiments show that HyCache achieves up to 7X higher throughput and 76X higher IOPS than Linux Ext4 file system, and can accelerate HDFS by 28% at 32-node scales (compared to vanilla HDFS).Index Terms-distributed file systems, user level file systems, hybrid file systems, heterogeneous storage, SSD

show abstract

“…Swift: Swift [27,57] is a system for the rapid and reliable specification, execution, and management of large-scale science and engineering workflows on clusters, grids, supercomputers, and clouds [64]. It supports applications that execute many tasks coupled by disk-resident datasets.…”

Section: Matrixmentioning

confidence: 99%

ZHT: A Light-Weight Reliable Persistent Dynamic Scalable Zero-Hop Distributed Hash Table

Zhou

Brandstatter

et al. 2013

2013 IEEE 27th International Symposium on Parallel and Distributed Processing

Self Cite

116

View full text Add to dashboard Cite

Abstract-This paper presents ZHT, a zero-hop distributed hash table, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. The goals of ZHT are delivering high availability, good fault tolerance, high throughput, and low latencies, at extreme scales of millions of nodes. ZHT has some important properties, such as being light-weight, dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append (providing lock-free concurrent key/value modifications) in addition to insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 512-cores, to an IBM Blue Gene/P supercomputer with 160K-cores. Using micro-benchmarks, we scaled ZHT up to 32K-cores with latencies of only 1.1ms and 18M operations/sec throughput. This work provides three real systems that have integrated with ZHT, and evaluate them at modest scales. 1) ZHT was used in the FusionFS distributed file system to deliver distributed meta-data management at over 60K operations (e.g. file create) per second at 2K-core scales. 2) ZHT was used in the IStore, an information dispersal algorithm enabled distributed object storage system, to manage chunk locations, delivering more than 500 chunks/sec at 32-nodes scales. 3) ZHT was also used as a building block to MATRIX, a distributed job scheduling system, delivering 5000 jobs/sec throughputs at 2K-core scales. We compared ZHT against other distributed hash tables and key/value stores and found it offers superior performance for the features and portability it supports.

show abstract

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Cited by 81 publications

References 23 publications

Comparing FutureGrid, Amazon EC2, and Open Science Grid for Scientific Workflows

Comparing FutureGrid, Amazon EC2, and Open Science Grid for Scientific Workflows

HyCache: A User-Level Caching Middleware for Distributed File Systems

ZHT: A Light-Weight Reliable Persistent Dynamic Scalable Zero-Hop Distributed Hash Table

Contact Info

Product

Resources

About