Minimizing Communication in All-Pairs Shortest Paths

Abstract

Abstract-We consider distributed memory algorithms for the all-pairs shortest paths (APSP) problem. Scaling the APSP problem to high concurrencies requires both minimizing inter-processor communication as well as maximizing temporal data locality. The 2.5D APSP algorithm, which is based on the divide-andconquer paradigm, satisfies both of these requirements: it can utilize any extra available memory to perform asymptotically less communication, and it is rich in semiring matrix multiplications, which have high temporal locality. We start by introducing a block-cyclic 2D (minimal memory) APSP algorithm. With a careful choice of block-size, this algorithm achieves known communication lower-bounds for latency and bandwidth. We extend this 2D block-cyclic algorithm to a 2.5D algorithm, which can use c extra copies of data to reduce the bandwidth cost by a factor of c 1/2 , compared to its 2D counterpart. However, the 2.5D algorithm increases the latency cost by c 1/2 . We provide a tighter lower bound on latency, which dictates that the latency overhead is necessary to reduce bandwidth along the critical path of execution. Our implementation achieves impressive performance and scaling to 24,576 cores of a Cray XE6 supercomputer by utilizing well-tuned intra-node kernels within the distributed memory algorithm.

Solving All-Pairs Shortest-Paths Problem in Large Graphs Using Apache Spark

“…In [19], Solomonik et al propose recursive 2D block-cyclic algorithm that achieves lower-bound on communication latency and bandwidth, which they next extend to 2.5D communication avoiding formulation. On a machine with 24,576 cores, the algorithms maintain strong scaling for problems with n = 32768 nodes, and weak scaling for up to n = 131072 nodes.…”

“…The paper is also noteworthy for its extensive review of distributed memory APSP solvers. The advantage of the recursive formulation is induced data locality, which directly contributes to improved performance [19]. However, in Spark, the concept of data locality is much weaker, since Spark's runtime system has a significant freedom in scheduling where to materialize or move data for computations.…”

“…The standard approach to handle APSP in such cases is to use a variant of classic Floyd-Warshall [5] or Johnson algorithms [5], with complexity O(|V | 3 ) and O(|V ||E| + |V | 2 log(|V |)), respectively. The Johnson algorithm offers better asymptotic behaviour for reasonably sparse graphs, but typically Floyd-Warshall derivatives outperform it as they allow for better computational density (see also [19]). When using Floyd-Warshall and related algorithms, we will represent adjacency matrix of G by A, where A i j = A ji = w i j stores the weight of edge between vertices with indices i and j.…”

“…The second method, abbreviated as DC-GbE, is highly optimized divide-and-conquer solver by Solomonik et al [19], available from https://github.com/solomonik/APSP. The solver has been demonstrated to scale extremely well to very large parallel machines, and is the state-of-the-art HPC solution.…”

On Characterizing the Data Access Complexity of Programs

“…Irony et al [21] used a geometric reasoning with the Loomis-Whitney inequality [23] to present an alternate proof to Hong and Kung's [20] for I/O lower bounds on standard matrix multiplication. More recently, Demmel's group at UC Berkeley has developed lower bounds as well as optimal algorithms for several linear algebra computations including QR and LU decomposition and the all-pairs shortest paths problem [1,3,13,33].…”