All-Pairs: An Abstraction for Data-Intensive Computing on Campus Grids

Abstract

Abstract-Today, campus grids provide users with easy access to thousands of CPUs. However, it is not always easy for nonexpert users to harness these systems effectively. A large workload composed in what seems to be the obvious way by a naive user may accidentally abuse shared resources and achieve very poor performance. To address this problem, we argue that campus grids should provide end users with high-level abstractions that allow for the easy expression and efficient execution of data intensive workloads. We present one example of an abstraction -All-Pairs -that fits the needs of several applications in biometrics, bioinformatics, and data mining. We demonstrate that an optimized All-Pairs abstraction is both easier to use than the underlying system, achieves performance orders of magnitude better than the obvious but naive approach, and is both faster and more efficient than a tuned conventional approach. This abstraction has been in production use for one year on a 500-CPU campus grid at the University of Notre Dame, and has been used to carry out a groundbreaking analysis of biometric data.

A distributed computing framework for All-to-All comparison problems

“…All-pairs is presented by Moretti et al [15] for data-intensive computing on campus grids. It provides an abstraction to users to deal with All-to-All Comparison Problems.…”

“…Among the existing frameworks, Hadoop [14] is popularly used to support the MapReduce computation pattern, but is inefficient in processing of All-to-All Comparison Problems. All-pairs [15] is designed for All-to-All Comparison Problems in a Campus Grid, but its application range is limited due to its brute-force data storage strategy, which stores all the data on all the worker nodes.…”

Coordinating Computation and I/O in Massively Parallel Sequence Search

“…First, MapReduce only handles one-dimensional input and hence is not suitable for implementing both query segmentation and database segmentation approaches. Moretti et al has reported a similar observation that MapReduce is not sufficient to express all-to-all style computation [43]. The existing MapReduce BLAST implementation, i.e., CloudBLAST [44], only implements query segmentation and stores the entire database on each node.…”

Evaluating the suitability of mapreduce for surface temperature analysis codes

“…Porting of ccc-gistemp to other scalable systems intended for data-intensive computing such as Dryad [12], All-Pairs [17] and Pregel [16] would provide a comparative study of the various programming abstractions that are suitable. Likewise implementations of MapReduce which use existing highperformance shared filesystems are now available (e.g.…”