PANDA: Extreme Scale Parallel K-Nearest Neighbor on Distributed Architectures

Abstract

Abstract-Computing k-Nearest Neighbors (KNN) is one of the core kernels used in many machine learning, data mining and scientific computing applications. Although kd-tree based O(log n) algorithms have been proposed for computing KNN, due to its inherent sequentiality, linear algorithms are being used in practice. This limits the applicability of such methods to millions of data points, with limited scalability for Big Data analytics challenges in the scientific domain. In this paper, we present parallel and highly optimized kdtree based KNN algorithms (both construction and querying) suitable for distributed architectures. Our algorithm includes novel approaches for pruning search space and improving load balancing and partitioning among nodes and threads. Using TB-sized datasets from three science applications: astrophysics, plasma physics, and particle physics, we show that our implementation can construct kd-tree of 189 billion particles in 48 seconds on utilizing ∼50,000 cores. We also demonstrate computation of KNN of 19 billion queries in 12 seconds. We demonstrate almost linear speedup both for shared and distributed memory computers. Our algorithms outperforms earlier implementations by more than order of magnitude; thereby radically improving the applicability of our implementation to state-of-the-art Big Data analytics problems. In addition, we showcase performance and scalability on the recently released Intel R Xeon Phi TM processor showing that our algorithm scales well even on massively parallel architectures.

Multilevel graph embedding

“…These edges may be considered (by taking to always be the distance between aggregates) but doing so is computationally infeasible for large graphs. One potential solution to this problem would be to use efficient k ‐nearest neighbor methods (e.g., Reference 15) to locate geometrically close unconnected aggregates and use such information to ensure that their balls will not overlap.…”

ArborX

“…A typical implementation of nearest traversal uses a priority queue based on distances, using the closest node in each iteration. An alternative and better performing approach, first derived for k-d trees in Patwary et al [2016], is to use a stack. As stack is a Last-In-First-Out data structure, it is possible to get a behavior similar to the one of a priority queue by adding a child with a shorter distance second (so that it sits on top of the stack).…”

Ethnic music exploration guided by personalized recommendations: system design and evaluation

“…The source code for the recommendation system is available on GitHub. 2 The proposed algorithm relies only on minimum Euclidean distances, which can be computed efficiently using distributed tree structures [16,17] or gpu-based implementations [18]. The proposed system is thus horizontally scalable and suitable for distributed applications.…”

“…Users explore different paths in the space by "liking" or "skipping" tracks. As both the mapping and the search processes are amenable to distributed tree [16,17] and gpu-based [18] parallel searches, the proposed system could be scaled up to accommodate increasing music collections and user bases.…”