A Task Pipelining Framework for e-Science Workflow Management Systems

Kim, H.S.; Cho, In Soon; Yeom, Heon Y.

doi:10.1109/ccgrid.2008.47

Cited by 6 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Classically, SWMSs have been envisioned to enable in software the 'analytic workflow' experienced by scientists-the process by which data is processed and analyzed to support exploration and understanding of an underlying model system [58,38]. Since much of current scientific research necessitates understanding multidimensional datasets with requisite computational processing, SWMSs have been presumed to serve as an 'enabling technology' by which intellectual advancement can be accelerated.…”

Section: Challenges In Data-intensive Sciencementioning

confidence: 99%

Contemporary challenges for data-intensive scientific workflow management systems

Mork

Martin

Zhao

2015

Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science

View full text Add to dashboard Cite

Data-intensive sciences now represent the forefront of current scientific computing. To handle this 'Big Data' focus, scientists demand enabling technologies that can adapt to the increasingly distributed, collaborative, and exploratory scientific milieu. However, how these challenges have changed the design requirements of scientific workflow management systems (SWMSs) has not been assessed. First, how scientists currently use SWMSs was determined through a comprehensive usage survey examining 1455 research publications from 2013 to July 31st, 2015. To understand how data-intensive scientists are producing impactful research, we further examined usage of two major research clouds, the Open Science Data Cloud (OSDC) and Cornell's Red Cloud. Here, we present a road map for SWMS development for data-intensive sciences. SWMSs are now needed that interconnect diverse software packages while enabling data exploration and multiuser interaction across distributed software and hardware environments.

show abstract

Section: Challenges In Data-intensive Sciencementioning

confidence: 99%

Contemporary challenges for data-intensive scientific workflow management systems

Mork

Martin

Zhao

2015

Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science

View full text Add to dashboard Cite

show abstract

“…Our system is distinguished from VLAM-G in that scheduling strategies of legacy workflow managers can be easily integrated. In [2], a storage layer for e-science workflow management systems, called PFS is designed. PFS makes it easy to integrate IO layer with workflow scheduler through triggering interface.…”

Section: Storage Layermentioning

confidence: 99%

“…It also provides a general framework independently of the workflow management systems being used. The storage layer can be described in a workflow specification and thus, a user is able to construct a task pipelining framework without any further efforts except presenting a workflow specification for the Pipe File System(PFS) [2]. Second phase is to have a job scheduler by utilizing the storage layer so that the total execution time will be reduced by maximizing the concurrency of adjacent jobs.…”

mentioning

confidence: 99%

“…First phase is to design a storage layer which helps the workflow manager to schedule several parallel jobs concurrently. A simplified task pipelining framework for workflow systems was developed in [2] and it is a simple distributed file system that supports various legacy applications without modification to the existing applications. It also provides a general framework independently of the workflow management systems being used.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Design of Grid Workflow System Scheduler for Task Pipelining

Lee¹

2010

Journal of the Korea Society of Computer and Information

View full text Add to dashboard Cite

The power of computational Grid resources can be utilized on users desktop by employing workflow managers. It also helps scientists to conveniently put together and run their own scientific workflows. Generally, stage-in, process and stage-out are serially executed and workflow systems help automate this process. However, as the data size is exponentially increasing and more and more scientific workflows require multiple processing steps to obtain the desired output, we argue that the data movement will possess high portion of overall running time. In this paper, we improved staging time and design a new scheduler where the system can execute concurrently as many jobs as possible. Our simulation study shows that 10% to 40% improvement in running time can be achieved through our approach.

show abstract

Provenance security guarantee from origin up to now in the e-Science environment

Jung

Yeom

2011

Journal of Systems Architecture

View full text Add to dashboard Cite

A Task Pipelining Framework for e-Science Workflow Management Systems

Cited by 6 publications

References 13 publications

Contemporary challenges for data-intensive scientific workflow management systems

Contemporary challenges for data-intensive scientific workflow management systems

Design of Grid Workflow System Scheduler for Task Pipelining

Provenance security guarantee from origin up to now in the e-Science environment

Contact Info

Product

Resources

About