A Workflow-Aware Storage System: An Opportunity Study

Vairavanathan, Emalayan; Al-Kiswany, Samer; Costa, Lauro Beltrao; Zhang, Zhao; Katz, Daniel S.; Wilde, Michael; Ripeanu, Matei

doi:10.1109/ccgrid.2012.109

Cited by 24 publications

(18 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The details of workflow enactment depends on the precise data access semantics and API offered by this intermediate space. At one end of the spectrum are intermediate storage systems that offer a shared file-system with complete POSIX API support (e.g., MosaStore [1,21], and Chirp/Parrot [20]). Whereas, at the other end of the spectrum are two overlapping solutions: (1) solutions that offer a custom API, possibly specialized for classes of applications (e.g., the Hadoop File System -HDFS); and, (2) solutions that do not offer a shared name and storage space and leave explicit data movement between independent storage nodes and space management to be handled by the workflow scheduler.…”

Section: Storage Systemsmentioning

confidence: 99%

Evaluating storage systems for scientific data in the cloud

Maheshwari

Wozniak

Yang

et al. 2014

Proceedings of the 5th ACM Workshop on Scientific Cloud Computing

Self Cite

View full text Add to dashboard Cite

Infrastructure-as-a-Service (IaaS) clouds are an appealing resource for scientific computing. However, the bare-bones presentation of raw Linux virtual machines leaves much to the application developer. For many cloud applications, effective data handling is critical to efficient application execution. This paper investigates the capabilities of a variety of POSIX-accessible distributed storage systems to manage data access patterns resulting from workflow application executions in the cloud. We leverage the expressivity of the Swift parallel scripting framework to benchmark the performance of a number of storage systems using synthetic workloads and three real-world applications. We characterize two representative commercial storage systems (Amazon S3 and HDFS, respectively) and two emerging research-based storage systems (Chirp/Parrot and MosaStore). We find the use of aggregated node-local resources effective and economical compared with remotely located S3 storage. Our experiments show that applications run at scale with MosaStore show up to 30% improvement in makespan time compared with those run with S3. We also find that storage-system driven application deployments in the cloud results in better runtime performance compared with an on-demand datastaging driven approach.

show abstract

Section: Storage Systemsmentioning

confidence: 99%

Evaluating storage systems for scientific data in the cloud

Maheshwari

Wozniak

Yang

et al. 2014

Proceedings of the 5th ACM Workshop on Scientific Cloud Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…HDFS [6] exposes data locality through a customized remote procedure call to the Hadoop execution engine. MosaStore [1,30] takes the approach of embedding data locality in the extended attributes of the POSIX standard. Both approaches require that the execution engine process locality information in order to make it transparent to programmers.…”

Section: Related Workmentioning

confidence: 99%

Parallelizing the execution of sequential scripts

Zhang

Armstrong

Wozniak

et al. 2013

Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Self Cite

View full text Add to dashboard Cite

Scripting is often used in science to create applications via the composition of existing programs. Parallel scripting systems allow the creation of such applications, but each system introduces the need to adopt a somewhat specialized programming model. We present an alternative scripting approach, AMFS Shell, that lets programmers express parallel scripting applications via minor extensions to existing sequential scripting languages, such as Bash, and then execute them in-memory on large-scale computers. We define a small set of commands between the scripts and a parallel scripting runtime system, so that programmers can compose their scripts in a familiar scripting language. The underlying AMFS implements both collective (fast file movement) and functional (transformation based on content) file management. Tasks are handled by AMFS's built-in execution engine. AMFS Shell is expressive enough for a wide range of applications, and the framework can run such applications efficiently on large-scale computers.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. SC13

show abstract

“…Replica synchronization is done in the MDS of MosaStore. The Gfarm file system [10] the replication mechanism is used for data replication for the reliability and availability. In the distributed and parallel file system, the MDS controls the data replication and send the data to the storage servers; this makes pressure to the MDS.…”

Section: Related Workmentioning

confidence: 99%