PHiDJ: Parallel similarity self-join for high-dimensional vector data with MapReduce

Fries, Sergej; Boden, Brigitte; Stępień, G.; Seidl, Thomas

doi:10.1109/icde.2014.6816701

Cited by 33 publications

(33 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the cross aggregate ranking problem studied in this paper, there is no predicate defining part of the Cartesian product to be evaluated. As a result, query processing techniques for equi-join [31], thetajoin [32], similarity join [33], [34] are not applicable to our query processing problem.…”

Section: Mapreduce Variants and Enhancementsmentioning

confidence: 99%

“…The method uses the MapReduce shuffle mechanism to deliver tuples with the same join key to the same reducer. Studies on join processing in MapReduce are largely concerned with optimizing the query performance with respect to different join query types, such as equi-join [31], theta-join [32], similarity join [33], [34], k-NN join [35] and top-k join [28].…”

Section: Mapreduce Variants and Enhancementsmentioning

confidence: 99%

See 1 more Smart Citation

Entropy-Based Scheduling Policy for Cross Aggregate Ranking Workloads

Dai

Nutanong

Chow

et al. 2018

IEEE Trans. Serv. Comput.

View full text Add to dashboard Cite

Many data exploration applications require the ability to identify the top-k results according to a scoring function. We study a class of top-k ranking problems where top-k candidates in a dataset are scored with the assistance of another set. We call this class of workloads cross aggregate ranking. Example computation problems include evaluating the Hausdorff distance between two datasets, finding the medoid or radius within one dataset, and finding the closest or farthest pair between two datasets. In this paper, we propose a parallel and distributed solution to process cross aggregate ranking workloads. Our solution subdivides the aggregate score computation of each candidate into tasks while constantly maintains the tentative top-k results as an uncertain top-k result set. The crux of our proposed approach lies in our entropy-based scheduling technique to determine result-yielding tasks based on their abilities to reduce the uncertainty of the tentative result set. Experimental results show that our proposed approach consistently outperforms the best existing one in two different types of cross aggregate rank workloads using real datasets.Index Terms-Query processing, Knowledge and data engineering tools and techniques.

show abstract

Section: Mapreduce Variants and Enhancementsmentioning

confidence: 99%

Section: Mapreduce Variants and Enhancementsmentioning

confidence: 99%

Entropy-Based Scheduling Policy for Cross Aggregate Ranking Workloads

Dai

Nutanong

Chow

et al. 2018

IEEE Trans. Serv. Comput.

View full text Add to dashboard Cite

show abstract

“…To deal with the similarity joins problem efficiently on large-scale data set, many researchers try to propose the parallel algorithms based on MapReduce framework 10 for different data types, such as set data, [11][12][13][14][15] vector data 9,[16][17][18][19][20][21][22] and spatial data. 23 Lin et al 11 firstly proposed brute force and indexed approaches to implement pairwise document similarity comparisons with MapReduce, but many duplicated comparisons existed in such approaches.…”

Section: Similarity Join Query In Distributed Processing Modelmentioning

confidence: 99%

A novel approach for high‐dimensional vector similarity join query

Jia

Zhang

2016

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARY In this paper, we mainly focus on similarity joins on massive high‐dimensional vectors, it is a costly operation because of curse of dimensionality. The main idea of our proposed solution is to design a novel filtering method that can filter out as many vector pairs as possible at relative low cost. By using the good dimension reduction ability of piecewise aggregate approximation and symbolic aggregate approximation techniques, we proposed three novel approaches to deal with high dimensional‐vector similarity join query: single‐PAA–based vector similarity join query, multi‐PAA–based vector similarity join query and SAX‐based vector similarity join query. We conducted comprehensive experiments to test the performance of the above approaches, we also test the speedup ratio compared with the naive method: block nested loop join. The experimental results show that our approaches have much better performance than that of block nested loop join and also have good scalability. To the best of our knowledge, this is the first work to try to deal with high‐dimensional vector similarity joins using piecewise aggregate approximation and symbolic aggregate approximation techniques, and the approaches proposed in this paper provide a new way to process the massive high‐dimensional vector data set.

show abstract

“…The StatStream system [25] specializes in discovering correlations using a grid structure, but it incurs prohibitive communication cost in a distributed environment. Recently, partitioning-based approaches have attracted attention for distributed batch data processing [7], [19], [22]. However, such approaches are data-dependent and need an aprori data pre-scanning step to estimate the data distribution.…”

Section: Related Workmentioning

confidence: 99%

“…This has led to the development of many distributed, fault-tolerant, and realtime computation systems [2], [3], [13], [24]. Analogous to the trend observed in map-reduce systems (e.g., Apache Hadoop); where efficiently performing complex joins using map-reduce was a challenging problem [7], [19], [22], using distributed real-time computation engines for efficiently and continuously mining meaningful information from time-series is becoming challenging, as we will see later.…”

Section: Introductionmentioning

confidence: 99%

SigCO: Mining significant correlations via a distributed real-time computation engine

Guo

Calbimonte

Zhuang

et al. 2015

2015 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Abstract-The dramatic rise of time-series data produced in a variety of contexts, such as stock markets, mobile sensing, sensor networks, data centre monitoring, etc., has fuelled the development of large-scale distributed real-time computation systems (e.g., Apache Storm, Samza, Spark Streaming, S4, etc.). However, it is still unclear how certain time series mining tasks could be performed using such new emerging systems. In this paper, we focus on the task of efficiently discovering statistically significant correlations among a large number of time series via a distributed realtime computation engine. We propose a framework referred to as SigCO. In SigCO, we put forward a novel partition-aware data shuffling, which is able to adaptively shuffle time series data only to the relevant nodes of the distributed real-time computation engine. On the other hand, in SigCO we design a δ-hypercube structure based correlation computation approach which is capable of pruning unnecessary correlation computations. Finally, our extensive experimental evaluations on real and synthetic datasets establish that SigCO outperforms the baseline approaches in terms of diverse performance metrics.

show abstract

PHiDJ: Parallel similarity self-join for high-dimensional vector data with MapReduce

Cited by 33 publications

References 21 publications

Entropy-Based Scheduling Policy for Cross Aggregate Ranking Workloads

Entropy-Based Scheduling Policy for Cross Aggregate Ranking Workloads

A novel approach for high‐dimensional vector similarity join query

SigCO: Mining significant correlations via a distributed real-time computation engine

Contact Info

Product

Resources

About