Design and implementation of GXP make — A workflow system based on make

Taura, Kenjiro; Matsuzaki, Takashi; Misawa, Miwa; Kamoshida, Yoshikazu; Yokoyama, Daisuke; Dun, Nan; Shibata, Takeshi; Jun, Choi Sung; Tsujii, Jun’ichi

doi:10.1016/j.future.2011.05.026

Cited by 16 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since all these three systems do not provide any language to describe the dependencies of components/jobs, we generally perform each single job using them and leave the creation of the whole workflows to a known workflow engine called GXP Make [2]. GXP Make uses make to describe the workflow and provides the parallelization of tasks across clusters.…”

Section: Text Processing Workflowsmentioning

confidence: 99%

See 1 more Smart Citation

A Comparative Study of Data Processing Approaches for Text Processing Workflows

Chen

Taura

2012

2012 SC Companion: High Performance Computing, Networking Storage and Analysis

Self Cite

View full text Add to dashboard Cite

Abstract-Workflows are widely used in data-intensive applications since it facilities the composition of individual executables or scripts, providing an easy-to-use parallelization to domain experts. With considerable popularity of MapReduce framework, some researchers start to develop MapReduce-enabled workflows instead of general file-based ones. Meanwhile, being commercially available for nearly two decades for large-scale data processing, parallel database systems have also gotten wide attention in the support of workflows. This paper studies three realworld text processing workflows and develops them on top of several different large data processing approaches including an open source MapReduce implementation -Hadoop, a workfloworiented parallel database system -ParaLite, and a hybrid of MapReduce and parallel DBMS -Hive. We discuss their strength-/weaknesses both in terms of programmability and performance for each workflow. Our experiences and experimental results reveal some interesting trade-offs: (1) High-level query languages (SQL of ParaLite and HiveQL of Hive) are helpful for expressing data selection, aggregation and calculation by typical executables; (2) To reuse existing NLP tools, it is often important to be able to track the association between a document and its annotation attached by the tool, for which the expressiveness of SQL is particularly useful; (3) Each system has similar performance in the execution of overall workflows because essentially performing executables takes most of the time, but some small differences could reveal some potential trade-offs that each system entails for workflows.

show abstract

Section: Text Processing Workflowsmentioning

confidence: 99%

“…Many systems are proposed to execute workflows, including GXP Make [2], Swift [3], Pegasus [4] and Taverna [5]. A workflow is generally a DAG with many data processing tasks and their dependencies.…”

Section: Introductionmentioning

confidence: 99%

A Comparative Study of Data Processing Approaches for Text Processing Workflows

Chen

Taura

2012

2012 SC Companion: High Performance Computing, Networking Storage and Analysis

Self Cite

View full text Add to dashboard Cite

show abstract

“…While our work has been carried out using the IPS framework's many-task computing environment, the same approach could, at least in principle, be implemented in any many-task environment with the capability to express dependencies among tasks. Relying on literature reports, it appears that many of the frameworks previously discussed in a many-task context could express a dependency-driven formulation of parareal, including Uintah [17], Kepler [15], Taverna [20], Triana [4], GXP Make [25], Swift [27], Pegasus [6], Nephele [26], and PLASMA [24] however such publications rarely provide sufficient detail to determine whether, and how effectively, the system can handle the dynamic nature of the dependency-driven parareal, including convergence rates that can vary significantly by slice, or the more complex "windowed" variant of the algorithm described in Sec. 6.…”

Section: Related Workmentioning

confidence: 99%

A dependency-driven formulation of parareal

Elwasif

Foley

Bernholdt

et al. 2011

Proceedings of the 2011 ACM International Workshop on Many Task Computing on Grids and Supercomputers

View full text Add to dashboard Cite

Parareal is a novel algorithm that allows the solution of time-dependent systems of differential or partial differential equations (PDE) to be parallelized in the temporal domain. Parareal-based implementations of PDE problems can take advantage of this parallelism to significantly reduce the time to solution for a simulation (though at an increased total cost) while making effective use of the much larger processor counts available on current high-end systems. In this paper, we present a dynamic, dependency-driven version of the parareal algorithm which breaks the final sequential bottleneck remaining in the original formulation, making it amenable to a "many-task" treatment. We further improve the cost and execution time of the algorithm by introducing a moving window for time slices, which avoids the execution of tasks which contribute little to the final global solution. We describe how this approach has been realized in the Integrated Plasma Simulator (IPS), a framework for coupled multiphysics simulations, and examine the tradeoffs among time-to-solution, total cost, and resource utilization efficiency as a function of the compute resources applied to the problem.

show abstract

“…In addition, as data are stored in files, it is tedious to select a subset of data due to the difficulties in creating index on data. Nowadays, many systems are proposed to execute workflows, including GXP Make [4], Swift [5] and Taverna [6].…”

Section: Introductionmentioning

confidence: 99%

ParaLite: A Parallel Database System for Data-Intensive Workflows

Chen

Taura

2014

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

SUMMARYTo better support data-intensive workflows which are typically built out of various independently developed executables, this paper proposes extensions to parallel database systems called User-Defined eXecutables (UDX) and collective queries. UDX facilitates the description of workflows by enabling seamless integrations of external executables into SQL statements without any efforts to write programs confirming to strict specifications of databases. A collective query is an SQL query whose results are distributed to multiple clients and then processed by them in parallel, using arbitrary UDX. It provides efficient parallelization of executables through the data transfer optimization algorithms that distribute query results to multiple clients, taking both communication cost and computational loads into account. We implement this concept in a system called ParaLite, a parallel database system based on a popular lightweight database SQLite. Our experiments show that ParaLite has several times higher performance over Hive for typical SQL tasks and has 10x speedup compared to a commercial DBMS for executables. In addition, this paper studies a real-world text processing workflow and builds it on top of ParaLite, Hadoop, Hive and general files. Our experiences indicate that ParaLite outperforms other systems in both productivity and performance for the workflow. key words: data-intensive workflow, parallel database system, userdefined executable, collective query

show abstract

Design and implementation of GXP make — A workflow system based on make

Cited by 16 publications

References 23 publications

A Comparative Study of Data Processing Approaches for Text Processing Workflows

A Comparative Study of Data Processing Approaches for Text Processing Workflows

A dependency-driven formulation of parareal

ParaLite: A Parallel Database System for Data-Intensive Workflows

Contact Info

Product

Resources

About