Makeflow

Albrecht, Michael; Donnelly, Patrick J.; Bui, Peter; Thain, Douglas

doi:10.1145/2443416.2443417

Cited by 125 publications

(25 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Makeflow [5] is a command line workflow engine used to execute data-intensive scientific applications on a variety of distributed execution systems including campus clusters, clouds, and grids. The end user expresses a workflow using a syntax similar to Make.…”

Section: Classification Of Workflow Management Systemsmentioning

confidence: 99%

“…In recent years, numerous workflow management systems (WMSs) have been developed to manage the execution of diverse workflows on heterogeneous computing resources [3,4,5,6,7,8,9]. As user communities adopt and evolve WMSs to fit their own needs, many of the features and capabilities that were once common to most WMSs have become too distinct to share across systems.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

126

View full text Add to dashboard Cite

Automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today's computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. The paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

show abstract

Section: Classification Of Workflow Management Systemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

126

View full text Add to dashboard Cite

show abstract

“…Some of the well-known workflow management systems include Pegasus [5,26], HTCondor DAGMan [27,28], Taverna [4], Triana [29] and makeflow [30]. Pegasus, in particular, keeps the separation of workflow description and system environment description.…”

Section: Workflow Managementmentioning

confidence: 99%

Workflow performance improvement using model-based scheduling over multiple clusters and clouds

Maheshwari

Jung

Meng

et al. 2016

Future Generation Computer Systems

View full text Add to dashboard Cite

Please cite this article as: K. Maheshwari, E.-S. Jung, J. Meng, V. Morozov, V. Vishwanath, R. Kettimuthu, Workflow performance improvement using model-based scheduling over multiple clusters and clouds, Future Generation Computer Systems (2015), http://dx. AbstractIn recent years, a variety of computational sites and resources have emerged, and users often have access to multiple resources that are distributed. These sites are heterogeneous in nature and performance of different tasks in a workflow varies from one site to another. Additionally, users typically have a limited resource allocation at each site capped by administrative policies. In such cases, judicious scheduling strategy is required in order to map tasks in the workflow to resources so that the workload is balanced among sites and the overhead is minimized in data transfer. Most existing systems either run the entire workflow in a single site or use naïve approaches to distribute the tasks across sites or leave it to the user to optimize the allocation of tasks to distributed resources. This results in a significant loss in productivity. We propose a multi-site workflow scheduling technique that uses performance models to predict the execution time on resources and dynamic probes to identify the achievable network throughput between sites. We evaluate our approach using real world applications using the Swift parallel and distributed execution framework. We use two distinct computational environments-geographically distributed multiple clusters and multiple clouds. We show that our approach improves the resource utilization and reduces execution time when compared to the default schedule.

show abstract

“…Popular workflow management systems include Condor DAGMan [10], Pegasus [11], Taverna [12], and makeflow [13]. Regarding massively parallel job scheduling, Pegasus [11] provides a task clustering technique that groups parallel jobs based on fixed parameters such as number of groups.…”

Section: B Distributed Parallel Computingmentioning

confidence: 99%

“…Equation 13 can be further analyzed through the denominator max function. Figure 4 shows the plots of two terms, m × (DSI +DSO) Bshared (= t SI/SO (m)) and t EX (m), in the max function.…”

Section: Coscheduling With Networkmentioning

confidence: 99%

Pipelining/Overlapping Data Transfer for Distributed Data-Intensive Job Execution

Jung

Maheshwari

Kettimuthu

2013

2013 42nd International Conference on Parallel Processing

View full text Add to dashboard Cite

Scientific workflows are increasingly gaining attention as both data and compute resources are getting bigger, heterogeneous, and distributed. Many scientific workflows are both compute intensive and data intensive and use distributed resources. This situation poses significant challenges in terms of real-time remote analysis and dissemination of massive datasets to scientists across the community. These challenges will be exacerbated in the exascale era.Parallel jobs in scientific workflows are common, and such parallelism can be exploited by scheduling parallel jobs among multiple execution sites for enhanced performance. Previous scheduling algorithms such as heterogeneous earliest finish time (HEFT) did not focus on scheduling thousands of jobs often seen in contemporary applications. Some techniques, such as task clustering, have been proposed to reduce the overhead of scheduling a large number of jobs. However, scheduling massively parallel jobs in distributed environments poses new challenges as data movement becomes a nontrivial factor.We propose efficient parallel execution models through pipelined execution of data transfer, incorporating network bandwidth and reserved resources at an execution site. We formally analyze those models and suggest the best model with the optimal degree of parallelism. We implement our model in the Swift parallel scripting paradigm using GridFTP. Experiments on real distributed computing resources show that our model with optimal degrees of parallelism outperform the current parallel execution model by as much as 50% reduction of total execution time.

show abstract

Makeflow

Cited by 125 publications

References 22 publications

A characterization of workflow management systems for extreme-scale applications

A characterization of workflow management systems for extreme-scale applications

Workflow performance improvement using model-based scheduling over multiple clusters and clouds

Pipelining/Overlapping Data Transfer for Distributed Data-Intensive Job Execution

Contact Info

Product

Resources

About