MR‐search: massively parallel heuristic search

Abstract

MR-Search is a framework for massively parallel heuristic search. Based on the MapReduce paradigm, it efficiently utilizes all available resources: processors, memories, and disks. MR-Search uses OpenMP on shared memory systems, Message Passing Interface on clusters with distributed memory, and a combination of both on clusters with multi-core processors. Large graphs that do not fit into the main memory can be efficiently processed with an out-of-core variant. We implemented two node expansion strategies in MR-Search: breadth-first frontier search and breadth-first iterative deepening A*. With breadth-first frontier search, we computed large and powerful table-driven heuristics, so-called pattern databases that exceed the main memory capacity. These pattern databases were then used to solve random instances of the 24puzzle with breadth-first iterative deepening A* on systems with up to 4093 processor cores. MR-Search is conceptually simple. It takes care of data partitioning, process scheduling, out-of-core data merging, communication, and synchronization. Application developers benefit from the parallel computational capacity without having the burden of implementing parallel application code.The exponential growth of the search space has motivated researchers to devise parallel schemes like tree decomposition, work stealing [1, 2], or a combination of both [3]. However, all of these Solving a puzzle took from 7 to 24 min depending on the number of cores and the size of the problem. The largest puzzle took 18 min on 4093 nodes, expanding 1 billion nodes per second. Figure 9 shows the execution profiles. From bottom to top, the bar segments give the time spent in the map phase, the cost of load balancing after the map, the shuffle, the sort (group-by), the reduce,

Special Issue: MapReduce and its Applications

“…The paper by Urbani et al presents a scalable application of MapReduce to the problem of RDF data compression . The paper by Schütt et al presents MR‐Search, a framework for massively parallel heuristic search .…”