Adding Result Diversification to $$k$$NN-Based Joins in a Map-Reduce Framework

Souza, Vinícius; Carvalho, Luiz Olmes; de Oliveira, Daniel; Bedo, Marcos; Santos, Lúcio F. D.

doi:10.1007/978-3-031-39847-6_5

Cited by 2 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The work in [46] fulfils the gap of adding result diversification to kNN joins by means of Influence criteria which is a coverage-based criterion defined by the distance-based ternary relationship between each search reference, the dataset, and the result set in large-scale, big data frameworks. This paper solves the problem by extending the nested Better Results with Influence Diversification algorithm to a Map-Reduce framework.…”

Section: Related Workmentioning

confidence: 99%

Efficient Continuous kNN Join over Dynamic High-dimensional Data

Ukey,

Zhang,

Yang

et al. 2023

Preprint

View full text Add to dashboard Cite

Given a user dataset U and an object dataset I, a kNN join query in high-dimensional space returns the k nearest neighbors of each object in dataset U from the object dataset I. The kNN join is a basic and necessary operation in many applications, such as databases, data mining, computer vision, multi-media, machine learning, recommenda-tion systems, and many more. In the real world, datasets frequently update dynamically as objects are added or removed. In this paper, we propose novel methods of continuous kNN join over dynamic high-dimensional data. We firstly propose the HDR+ Tree, which supports more efficient insertion, deletion, and batch update. Further observed that the existing methods rely on globally correlated datasets for effec-tive dimensionality reduction, we then propose the HDR Forest. It clusters the dataset and constructs multiple HDR Trees to capture local correlations among the data. As a result, our HDR Forest is able to process non-globally correlated datasets efficiently. Two novel optimisations are applied to the proposed HDR Forest, including the precomputation of the PCA states of data items and pruning-based kNN recomputation during item deletion. For the completeness of the work, we also present the proof of computing distances in reduced dimensions of PCA in HDR Tree. Extensive experiments on real-world datasets show that the proposed methods and optimisations outperform the baseline algorithms of naive RkNN join and HDR Tree.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient Continuous kNN Join over Dynamic High-dimensional Data

Ukey,

Zhang,

Yang

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Efficient continuous kNN join over dynamic high-dimensional data

Ukey,

Zhang,

Yang

et al. 2023

World Wide Web

View full text Add to dashboard Cite

Given a user dataset $$\varvec{U}$$ U and an object dataset $$\varvec{I}$$ I , a kNN join query in high-dimensional space returns the $$\varvec{k}$$ k nearest neighbors of each object in dataset $$\varvec{U}$$ U from the object dataset $$\varvec{I}$$ I . The kNN join is a basic and necessary operation in many applications, such as databases, data mining, computer vision, multi-media, machine learning, recommendation systems, and many more. In the real world, datasets frequently update dynamically as objects are added or removed. In this paper, we propose novel methods of continuous kNN join over dynamic high-dimensional data. We firstly propose the HDR$$^+$$ + Tree, which supports more efficient insertion, deletion, and batch update. Further observed that the existing methods rely on globally correlated datasets for effective dimensionality reduction, we then propose the HDR Forest. It clusters the dataset and constructs multiple HDR Trees to capture local correlations among the data. As a result, our HDR Forest is able to process non-globally correlated datasets efficiently. Two novel optimisations are applied to the proposed HDR Forest, including the precomputation of the PCA states of data items and pruning-based kNN recomputation during item deletion. For the completeness of the work, we also present the proof of computing distances in reduced dimensions of PCA in HDR Tree. Extensive experiments on real-world datasets show that the proposed methods and optimisations outperform the baseline algorithms of naive RkNN join and HDR Tree.

show abstract

Adding Result Diversification to $$k$$NN-Based Joins in a Map-Reduce Framework

Cited by 2 publications

References 16 publications

Efficient Continuous kNN Join over Dynamic High-dimensional Data

Efficient Continuous kNN Join over Dynamic High-dimensional Data

Efficient continuous kNN join over dynamic high-dimensional data

Contact Info

Product

Resources

About