Community Resources for Enabling Research in Distributed Scientific Workflows

Silva, Rafael Ferreira da; Chen, Weiwei; Juve, Gideon; Vahi, Karan; Deelman, Ewa

doi:10.1109/escience.2014.44

Cited by 63 publications

(35 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The three other datasets represent actual applications and have been generated with the Pegasus Workflow Generator [8]. We consider three different datasets, named LIGO, MONTAGE, and GENOME, each containing 20 graphs of 100 nodes.…”

Section: Simulation Resultsmentioning

confidence: 99%

Parallel Scheduling of DAGs under Memory Constraints

Marchal

Nagy

Simon

et al. 2018

2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

Scientific workflows are frequently modeled as Directed Acyclic Graphs (DAG) of tasks, which represent computational modules and their dependencies in the form of data produced by a task and used by another one. This formulation allows the use of runtime systems which dynamically allocate tasks onto the resources of increasingly complex computing platforms. However, for some workflows, such a dynamic schedule may run out of memory by exposing too much parallelism. This paper focuses on the problem of transforming such a DAG to prevent memory shortage, and concentrates on shared memory platforms. We first propose a simple model of DAGs which is expressive enough to emulate complex memory behaviors. We then exhibit a polynomial-time algorithm that computes the maximum peak memory of a DAG, that is, the maximum memory needed by any parallel schedule. We consider the problem of reducing this maximum peak memory to make it smaller than a given bound by adding new fictitious edges, while trying to minimize the critical path of the graph. After proving this problem NP-complete, we provide an ILP solution as well as several heuristic strategies that are thoroughly compared by simulation on synthetic DAGs modeling actual computational workflows. We show that on most instances we are able to decrease the maximum peak memory at the cost of a small increase in the critical path, thus with little impact on quality of the final parallel schedule. Key-words: Ordonnancement parallèle de DAGs sous contraintes mémoireRésumé : Les applications de calcul scientifique sont souvent modélisées par des graphes de tâches orientés acycliques (DAG), qui représentent les tâches de calcul et leurs dépendances, sous la forme de données produites par une tâche et utilisées par une autre. Cette formulation permet l'utilisation d'API qui allouent dynamiquement les tâches sur les ressources de plateformes de calcul hétérogènes de plus en plus complexes. Cependant, pour certaines applications, un tel ordonnancement dynamique peut manquer de mémoire en exploitant trop de parallélisme. Cet article porte sur le problème consistant à transformer un tel DAG pour empêcher toute pénurie de mémoire, en se concentrant sur les plateformes à mémoire partagée. On propose tout d'abord un modèle simple de graphe qui est assez expressif pour émuler des comportements mémoires complexes. On expose ensuite un algorithme polynomial qui calcule le pic mémoire maximum d'un DAG, qui représente la mémoire maximale requise par tout ordonnancement parallèle. On considère ensuite le problème consistant à réduire ce pic mémoire maximal pour qu'il devienne plus petit qu'une borne donnée en rajoutant des arêtes fictives, tout en essayant de minimiser le chemin critique du graphe. Après avoir prouvé ce problème NPcomplet, on fournit un programme linéaire en nombres entiers le résolvant, ainsi que plusieurs stratégies heuristiques qui sont minitieusement comparées-sur des graphes synthétiques modélisant des applications de calcul réelles. On montre que sur la plupar...

show abstract

Section: Simulation Resultsmentioning

confidence: 99%

Parallel Scheduling of DAGs under Memory Constraints

Marchal

Nagy

Simon

et al. 2018

2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

show abstract

“…We collected workflow execution traces [38], [59] (including overhead and task runtime information) from real runs of the 5 scientific workflow applications previously described. The traces are used to feed the Workflow Generator [60] toolkit to create synthetic workflows. The toolkit uses statistical data gathered from traces of actual scientific workflow executions to generate realistic, synthetic workflows that resemble the real applications.…”

Section: Experiments Conditionsmentioning

confidence: 99%

Dynamic and Fault-Tolerant Clustering for Scientific Workflows

Chen

Silva

Deelman

et al. 2016

IEEE Trans. Cloud Comput.

Self Cite

View full text Add to dashboard Cite

Task clustering has proven to be an effective method to reduce execution overhead and to improve the computational granularity of scientific workflow tasks executing on distributed resources. However, a job composed of multiple tasks may have a higher risk of suffering from failures than a single task job. In this paper, we conduct a theoretical analysis of the impact of transient failures on the runtime performance of scientific workflow executions. We propose a general task failure modeling framework that uses a Maximum Likelihood estimation-based parameter estimation process to model workflow performance. We further propose 3 fault-tolerant clustering strategies to improve the runtime performance of workflow executions in faulty execution environments. Experimental results show that failures can have significant impact on executions where task clustering policies are not fault-tolerant, and that our solutions yield makespan improvements in such scenarios. In addition, we propose a dynamic task clustering strategy to optimize the workflow's makespan by dynamically adjusting the clustering granularity when failures arise. A trace-based simulation of five real workflows shows that our dynamic method is able to adapt to unexpected behaviors, and yields better makespans when compared to static methods.

show abstract

“…What is more, the typical price for storing all of a workflow's files in the cloud for the duration of its execution is much smaller than the price for computing. For example, the Workflow Gallery [46] provides a sample Montage application consisting of 1000 tasks that executes in 3 hours 10 minutes and generates files with a total size of 4.2GiB. According to current Google Cloud pricing [2], it costs substantial more to rent standard VMs for the duration of computation ($0.1583 at $0.05 per hour) than to store all files for the same amount of time ($0.0001128 at $0.026 per GiB per month that consists of 730 hours).…”

Section: Storage and File Transfer Modelmentioning

confidence: 99%

“…We evaluated the algorithms using ensembles consisting of synthetic workflows from the Workflow Gallery [46]. We have selected workflows representing several different classes of applications [32].…”

Section: Evaluation Proceduresmentioning

confidence: 99%

Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds

Bryk¹,

Malawski

Juve³

et al. 2015

J Grid Computing

Self Cite

View full text Add to dashboard Cite

This paper focuses on data-intensive workflows and addresses the problem of scheduling workflow ensembles under cost and deadline constraints in Infrastructure as a Service (IaaS) clouds. Previous research in this area ignores file transfers between workflow tasks, which, as we show, often have a large impact on workflow ensemble execution. In this paper we propose and implement a simulation model for handling file transfers between tasks, featuring the ability to dynamically calculate bandwidth and supporting a configurable number of replicas, thus allowing us to simulate various levels of congestion. The resulting model is capable of representing a wide range of storage systems available on clouds: from inmemory caches (such as memcached), to distributed file systems (such as NFS servers) and cloud storage (such as Amazon S3 or Google Cloud Storage). We observe that file transfers may have a significant impact on ensemble execution; for some applications up to 90 % of the execution time is spent on file transfers. Next, we propose and evaluate a novel scheduling algorithm that minimizes the number of transfers by taking advantage of data caching and file locality. We find that for data-intensive applications it performs better than other scheduling algorithms. Additionally, we modify the original scheduling algorithms to effectively operate in environments where file transfers take non-zero time.

show abstract

Community Resources for Enabling Research in Distributed Scientific Workflows

Cited by 63 publications

References 30 publications

Parallel Scheduling of DAGs under Memory Constraints

Parallel Scheduling of DAGs under Memory Constraints

Dynamic and Fault-Tolerant Clustering for Scientific Workflows

Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds

Contact Info

Product

Resources

About