P&amp;#x2217;: A model of pilot-abstractions

Luckow, André; Santcroos, Mark; Merzky, André; Weidner, Ole; Mantha, Pradeep; Jha, Shantenu

doi:10.1109/escience.2012.6404423

Cited by 31 publications

(42 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The P* approach to pilots has the following natural advantages: (i) it permits late binding of workloads to resources and (ii) the decoupling of tasks from resource management can be extended to data. This extension of the P* model with Pilot-Data is described in [14].…”

Section: A the P* Modelmentioning

confidence: 99%

Exploring Dynamic Enactment of Scientific Workflows Using Pilot-Abstractions

Santcroos

Schaik

Shahand

et al. 2013

2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing

View full text Add to dashboard Cite

Current workflow abstractions in general lack: (a) an adequate approach to handle distributed data and (b) proper separation between logical tasks and data-flow from their mapping onto physical locations. As the complexity and dynamism of data and processing distribution have increased, optimized mapping of logical tasks to physical resources have become a necessity to avoid bottlenecks. We argue that the management of dynamic data and compute should become part of the runtime system of workflow engines to enable workflows to scale as necessary to address big data challenges and fully exploit distributed computing infrastructures (DCI). In this paper we explore how the P* model for pilot-abstractions, which proposes a clear separation between the logical compute and data units and their realization as a job or a file in some physical resource, could provide these capabilities for such a runtime environment. The Pilot-API provides a general-purpose interface to pilotabstractions and the ability to assign compute and data resources to them. We share our experience of using the case study of a DNA sequencing pipeline, to re-implement the workflow using the Pilot-API. This first exercise, which resulted in a running application that is discussed here, illustrates the potential of this API to address (a) and (b). Our initial results indicate that the pilot abstractions (as captured by the P* model) offer an interesting approach to explore the design of a new generation of workflow management systems and runtime environments that are capable of intelligently deciding on application-aware late binding to physical resources.

show abstract

Section: A the P* Modelmentioning

confidence: 99%

Exploring Dynamic Enactment of Scientific Workflows Using Pilot-Abstractions

Santcroos

Schaik

Shahand

et al. 2013

2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing

View full text Add to dashboard Cite

show abstract

“…On the other hand, user-level or application scheduling denotes the division of the calculation workload into tasks because is usually performed by user's tools. It is noteworthy to mention that some abstraction models of pilot jobs have been already proposed [14,32,33]. These abstractions and formalisms undoubtedly result adequate to describe scheduling algorithms based on pilots, but are succinct to detail the design of some frameworks.…”

Section: Overall Vision and Nomenclaturementioning

confidence: 99%

“…Conventional RPC protocols and bidirectional implementations are not recommended. The use of message accumulators also increases the complexity and lag times in communications [26,28,32]. However, the choice of a pull method can influence the scheduling algorithms selected, as will be explained later.…”

Section: Minimal Functionalitiesmentioning

confidence: 99%

“…However, it is difficult to guide provisioning if the pilot execution is delegated to an external GS [19,25,44]. Thus, to build a framework that manages tasks and provisioning together [26,28,32] is more feasible.…”

Section: Advanced Scheduling and Hierarchymentioning

confidence: 99%

“…Usually, staging files is considered as part of the internal operations of the system [33] or even a separate overhead in other works [32]. In this study transfer times are included in pilot overhead because the pilot downloads the needed inputs and uploads the outputs generated by a task.…”

Section: Turnaround and Controlled Overheadmentioning

confidence: 99%

See 2 more Smart Citations

GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs

Montero

Huedo

Castejón

et al. 2015

Future Generation Computer Systems

View full text Add to dashboard Cite

Current systems based on pilot jobs are not exploiting all the scheduling advantages that the technique offers, or they lack compatibility or adaptability. To overcome the limitations or drawbacks in existing approaches, this study presents a different general-purpose pilot system, GWpilot. This system provides individual users or institutions with a more easy-to-use, easy-toinstall, scalable, extendable, flexible and adjustable framework to efficiently run legacy applications. The framework is based on the GridWay meta-scheduler and incorporates the powerful features of this system, such as standard interfaces, fair-share policies, ranking, migration, accounting and compatibility with diverse infrastructures. GWpilot goes beyond establishing simple network overlays to overcome the waiting times in remote queues or to improve the reliability in task production. It properly tackles the characterisation problem in current infrastructures, allowing users to arbitrarily incorporate customised monitoring of resources and their running applications into the system. This functionality allows the new framework to implement innovative scheduling algorithms that accomplish the computational needs of a wide range of calculations faster and more efficiently. The system can also be easily stacked under other software layers, such as self-schedulers. The advanced techniques included by default in the framework result in significant performance improvements even when very short tasks are scheduled.

show abstract

On the complexities of utilizing large‐scale lightpath‐connected distributed cyberinfrastructure

Maassen

Werkhoven

Meersbergen

et al. 2016

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

EDITORIAL Report on the first workshop on negative and null results in eScience 1. INTRODUCTION New techniques and technologies, such as the use of large-scale computing, influence research approaches, methods, and scales and are rapidly changing the scientific landscape. Research projects in eScience thus start with many assumptions and many unknowns and are often complex. While the scientific process is sometimes viewed, at least in hindsight, as a linear progression from one good idea to the next, it is in fact fraught with false starts, wrong assumptions, and dead ends. The increasing reliance on computation adds to the scope of problems that occur. Researchers invest a significant amount of time and effort in their research. Funding agencies similarly make large investments to support such research, on the assumption that most of the research will be successful. When the research assumptions and hypotheses turn out to be false, causing results that are "negative" or "null", the natural bias is to judge that the research project "failed." The history of science, however, shows that negative results may be an opportunity to revolutionize a field of study. For example, Fleming noticed that his flu cultures were contaminated by mold, but that there was infection around that mold, leading to his discovery of Penicillin. Similarly, a project today may fail because of the misuse or failure of computational support. Such "failures" actually indicate that there is an opportunity for the cyberinfrastructure research community to improve computing resources and tools. The interaction of these modes of failure is multi-faceted. Negative results have been difficult to find in published papers in all scientific domains. We identify three reasons for this. First, negative results may not be identified as such but simply considered mistakes. Such cases may never be investigated further. Secondly, paper referees may demand a higher standard from such results, because they are more difficult to understand or challenge the conventional narrative. Third, researchers may self-select against publishing such results in light of the previous point. Our experience with and observations of numerous projects in the area of eScience inspired us to create a workshop as a forum to initiate the discussion of negative and null results in the context of eScience (Section 4). Our effort attempts to systematically approach problems voiced in the popular media [1, 2] about the importance of negative results. This paper encapsulates the topics presented and lessons learned at this workshop, including efforts to: (1) Examine the topics surrounding the notion of negative and null results. Classify and propose a taxonomy of negative and null results in eScience. (2) Understand the causes of negative results. Discuss the activities and actions undertaken in order to mitigate the effects of negative results. (3) Challenge confirmation bias and the prejudice against negative results. Consider how to present negative results to the scientific communi...

show abstract

P∗: A model of pilot-abstractions

Cited by 31 publications

References 24 publications

Exploring Dynamic Enactment of Scientific Workflows Using Pilot-Abstractions

Exploring Dynamic Enactment of Scientific Workflows Using Pilot-Abstractions

GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs

On the complexities of utilizing large‐scale lightpath‐connected distributed cyberinfrastructure

Contact Info

Product

Resources

About