Generate‐map‐reduce: An extension to map‐reduce to support shared data and recursive computations

Abstract

SUMMARYIt is difficult to express the parallelism present in complex computations by using existing higher level abstractions such as MapReduce and Dryad. These computations include applications from wide variety of domains, like Artificial Intelligence, Decision Tree Algorithms, Association Rule Mining, Recommender Systems, Graph Algorithms, Clustering Algorithms, Compute Intensive Scientific Workflows, Optimization Algorithms, and so forth. Their execution graphs introduce new challenges in terms of programmer expressibility and runtime performance such as iterative and recursive computations, shared communication model, and so forth. We propose an extension to MapReduce, called Generate-Map-Reduce (GMR), targeted towards modeling these applications. GMR introduces a new Generate abstraction into the MapReduce framework that captures recursive computations. The runtime also supports iterative jobs and a distributed communication model by using shared data structures. We illustrate recursive computations with GMR by modeling complex applications such as simulated annealing, A* search, and adaptive quadrature computation that require recursive spawning of new tasks to handle variable degree of parallelism. GMR runtime supports caching of common data across iterations in memory and local disks. We illustrate how this caching helps in achieving significant speedup for iterative computations by modeling k-means clustering. Copyright

Performance enhancement for iterative data computing with in‐memory concurrent processing

“…Dharanipragada et al proposed Generate‐Map‐Reduce (GMR), which was an extension to MapReduce, to support iterative jobs and a distributed communication model by using shared data structures. GMR captured recursive computations by modeling iterative applications, such as simulated annealing and A* search . The main idea is similar to Spark in‐memory processing langrage.…”

“…GMR captured recursive computations by modeling iterative applications, such as simulated annealing and A* search. 26 The main idea is similar to Spark in-memory processing langrage.…”

A new closed frequent itemset mining algorithm based on GPU and improved vertical structure

“…One of the most famous distributed techniques is the cloud computing . With MapReduce , almost each problem can be solved by using cloud computing.…”