Design and Implementation of GXP Make -- A Workflow System Based on Make

Abstract

This paper describes the rational behind designing workflow systems based on the Unix make by showing a number of idioms useful for workflows comprising many tasks. It also demonstrates a specific design and implementation of such a workflow system called GXP make. GXP make supports all the features of GNU make and extends its platforms from single node systems to clusters, clouds, supercomputers, and distributed systems. Interestingly, it is achieved by a very small code base that does not modify GNU make implementation at all. While being not ideal for performance, it achieved a useful performance and scalability of dispatching one million tasks in approximately 16,000 seconds (60 tasks per second, including dependence analysis) on an 8 core Intel Nehalem node. For real applications, recognition and classification of protein-protein interactions from biomedical texts on a supercomputer with more than 8,000 cores are described.

Harnessing the Power of Many: Extensible Toolkit for Scalable Ensemble Applications

“…Adapting these systems to user requirements is non-trivial due to their feature richness and end-to-end design. Ruffus [25], COSMOS [26], and GXP Make [27] limit the capabilities and prioritize interface simplicity. Galaxy [28], Taverna [29], BioPipe [30], and Copernicus [31] focus on providing tailored interfaces to domain scientists.…”

Bioinformatics Workflow Management With The Wobidisco Ecosystem

“…"Map-Reduce" is such a kind of abstraction). GXP Make [53] (based on "GXP Shell" [54]) is a fun (ab)use of GNU-make as it provides a shell mksh that intercepts make's calls to run distributed workflows. Biomake [55] is another extension of the venerable make tool by making it more programmable thanks a Prolog execution engine.…”

Swift: A language for distributed parallel scripting

“…Since 317 files were mapped (in lines [47][48], the loop will submit 317 instances of the application in parallel to the execution provider. These will execute with a degree of parallelism subject to available resources.…”

“…A few groups have been working on parallel and distributed versions of make [47,48]. These tools use the concept of "virtual data," where the user defines the processing by which data is created and then calls for the final data product.…”