Scalable and Resilient Workflow Executions on Production Distributed Computing Infrastructures

Balderrama, Javier Rojas; Truong-Huu, Tram; Montagnat, Johan

doi:10.1109/ispdc.2012.24

Cited by 6 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several approaches have been devised to manage these complexities [46][47][48][49][50][51][52][53][54] but one of the persistent issues is the increase of the implementation burden imposed on applications. For example, in spite of progress made by grid computing [55,56] to transparently integrate diverse resources, most of the requirements involving the coordination of task execution still reside with the applications [57][58][59].…”

Section: Technical Origins and Motivationsmentioning

confidence: 99%

See 1 more Smart Citation

A Comprehensive Perspective on Pilot-Job Systems

Turilli

Santcroos²,

Jha

2018

ACM Comput. Surv.

View full text Add to dashboard Cite

Pilot-Job systems play an important role in supporting distributed scientific computing. They are used to consume more than 700 million CPU hours a year by the Open Science Grid communities, and by processing up to 1 million jobs a day for the ATLAS experiment on the Worldwide LHC Computing Grid. With the increasing importance of task-level parallelism in high-performance computing, Pilot-Job systems are also witnessing an adoption beyond traditional domains. Notwithstanding the growing impact on scientific research, there is no agreement upon a definition of Pilot-Job system and no clear understanding of the underlying abstraction and paradigm. Pilot-Job implementations have proliferated with no shared best practices or open interfaces and little interoperability. Ultimately, this is hindering the realization of the full impact of Pilot-Jobs by limiting their robustness, portability, and maintainability. This paper offers a comprehensive analysis of Pilot-Job systems critically assessing their motivations, evolution, properties, and implementation. The three main contributions of this paper are: (i) an analysis of the motivations and evolution of Pilot-Job systems; (ii) an outline of the Pilot abstraction, its distinguishing logical components and functionalities, its terminology, and its architecture pattern; and (iii) the description of core and auxiliary properties of Pilot-Jobs systems and the analysis of seven exemplar Pilot-Job implementations. Together, these contributions illustrate the Pilot paradigm, its generality, and how it helps to address some challenges in distributed scientific computing.

show abstract

Section: Technical Origins and Motivationsmentioning

confidence: 99%

“…Once tasks have been bound to pilots, Pilot systems are said to implement multi-level scheduling [5,16,54] because they include scheduling onto the DCR as well as scheduling onto the pilots. Unfortunately, the term "level" in multi-level is left unspecified making unclear what is scheduled and when.…”

Section: Jobmentioning

confidence: 99%

A Comprehensive Perspective on Pilot-Job Systems

Turilli

Santcroos²,

Jha

2018

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…Its data-driven nature makes the enactment engine able to automatically distribute computing tasks with the highest asynchronicity level. The computing tasks are abstracted so as to remain as independent as possible from the execution platform [73]. Provenance and reasoning on dense workflow execution traces bridges the gap between enactment technicalities and the (high-level) scientific process [74].…”

Section: Galaxymentioning

confidence: 99%

Scientific workflows: Past, present and future

Atkinson

Gesing

Montagnat

et al. 2017

Future Generation Computer Systems

Self Cite

102

View full text Add to dashboard Cite

International audienceThis special issue and our editorial celebrate 10 years of progress with data-intensive or scientific workflows. There have been very substantial advances in the representation of workflows and in the engineering of workflow management systems (WMS). The creation and refinement stages are now well supported, with a significant improvement in usability. Improved abstraction supports cross-fertilisation between different workflow communities and consistent interpretation as WMS evolve. Through such re-engineering the WMS deliver much improved performance, significantly increased scale and sophisticated reliability mechanisms. Further improvement is anticipated from substantial advances in optimisation. We invited papers from those who have delivered these advances and selected 14 to represent today's achievements and representative plans for future progress. This editorial introduces those contributions with an overview and categorisation of the papers. Furthermore, it elucidates responses from a survey of major workflow systems, which provides evidence of substantial progress and a structured index of related papers. We conclude with suggestions on areas where further research and development is needed and offer a vision of future research directions

show abstract

“…We also evaluate the proposed algorithms using realisticapplication workflows: a medical image analysis [16], an Alzheimer's disease case study [15] and a protein structure annotation workflow [1]. For each time slot, we generate a number of requests for these three workflows.…”

Section: Performance With Real Application Workflowsmentioning

confidence: 99%

Mapping workflow resource requests for bandwidth efficiency in data centers

Girisagar

Truong-Huu

Gurusamy

2016

Proceedings of the 17th International Conference on Distributed Computing and Networking

View full text Add to dashboard Cite

Representing a large class of coarse-grained distributed applications, workflows require large computing and bandwidth resources for their execution. With specific resource requirements due to data precedence and time disjointness among workflow tasks, mapping workflow resource requests in data centers is a challenging problem for cloud providers. While existing approaches only focus on satisfying computing resources and ignore the impact of mapping on bandwidth usage, we consider both computing and network resources to improve bandwidth efficiency in data centers while guaranteeing the performance of users' applications. We first formulate an integer programming optimization model for the mapping problem that minimizes the bandwidth allocated to workflows. We then develop two algorithms namely Critical Path Workflow Mapping (CPWM) and Edge Priority Workflow Mapping (EPWM) to solve this problem. We evaluate CPWM and EPWM through comprehensive simulations. The results show that CPWM and EPWM significantly reduce the bandwidth allocated for a workflow request by up to 66% for random workflows and 80% for realisticapplication workflows compared to baseline algorithms.

show abstract

Scalable and Resilient Workflow Executions on Production Distributed Computing Infrastructures

Cited by 6 publications

References 25 publications

A Comprehensive Perspective on Pilot-Job Systems

A Comprehensive Perspective on Pilot-Job Systems

Scientific workflows: Past, present and future

Mapping workflow resource requests for bandwidth efficiency in data centers

Contact Info

Product

Resources

About