Efficient K-Nearest Neighbor Graph Construction Using MapReduce for Large-Scale Data Sets

Abstract

SUMMARYThis paper presents an efficient method using Hadoop MapReduce for constructing a K-nearest neighbor graph (K-NNG) from a large-scale data set. K-NNG has been utilized as a data structure for data analysis techniques in various applications. If we are to apply the techniques to a large-scale data set, it is desirable that we develop an efficient K-NNG construction method. We focus on NN-Descent, which is a recently proposed method that efficiently constructs an approximate K-NNG. NNDescent is implemented on a shared-memory system with OpenMP-based parallelization, and its extension for the Hadoop MapReduce framework is implied for a larger data set such that the shared-memory system is difficult to deal with. However, a simple extension for the Hadoop MapReduce framework is impractical since it requires extremely high system performance because of the high memory consumption and the low data transmission efficiency of MapReduce jobs. The proposed method relaxes the requirement by improving the MapReduce jobs, which employs an appropriate key-value pair format and an efficient sampling strategy. Experiments on large-scale data sets demonstrate that the proposed method both works efficiently and is scalable in terms of a data size, the number of machine nodes, and the graph structural parameter K.

Billion-Scale Similarity Search with GPUs

“…Performance is improved sublinearly (∼ 1.6× for m = 20, ∼ 1.7× for m = 40). For comparison, the largest k-NN graph construction we are aware of used a dataset comprising 36.5 million 384d vectors, which took a cluster of 128 CPU servers 108.7 hours of compute [45], using NN-Descent [15]. Note that NN-Descent could also build or refine the k-NN graph for the datasets we consider, but it has a large memory overhead over the graph storage, which is already 80 GB for Deep1B.…”

Cancelled: A New Efficient Resource Management Framework for Iterative MapReduce Processing in Large-Scale Data Analysis

“…When the caching data is used at the current execution stage, the caching file and the index file stored in the local file of the node are loaded (lines 8-10). Because no cached data exists for the first iteration, the data being configured as invariant data among map outcomes are stored in the caching file (lines [11][12][13][14]. At this time, the byte offset of the caching file is recorded into the index file for combining the caching data and map outcomes.…”

“…The maximum number of containers that can be allocated to each node is calculated by each node (lines 5-10). Depending on the type of each task, the containers are assigned to each node by using the usage status of node resources and the resource scheduling information (lines [11][12][13][14][15][16][17][18]. At this time, the existing policy of Hadoop is maintained as the container allocation policy.…”

“…Figure 14 shows a pseudo-code of the k-Means application using our add-on iterative framework APIs. By just adding our add-on iterative framework APIs into the main function, a user can reuse the existing iterative applications without the modification of user-defined map/reduce classes (lines [8][9][10][11][12][13][14][15]. Therefore, our framework does not require the re-creation of the existing iterative applications.…”

“…Thirdly, we devise an iterative resource management technique which can allocate resource uniformly to every node in a Hadoop cluster. For this, we store iteration information into a meta-data table in HDFS (Hadoop Distributed File System) [13]. Fourthly, we devise a stop condition check mechanism for preventing the unnecessary computation by comparing the current iteration output with the previous iteration output.…”