Efficient Processing of Recursive Joins on Large-Scale Datasets in Spark

Phan, Thuong-Cang; Phan, Anh-Cang; Tran, Thi-To-Quyen; Trieu, Thanh-Ngoan

doi:10.1007/978-3-030-38364-0_35

Cited by 1 publication

(1 citation statement)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A recent study [24] has proposed a solution to optimize recursive join in Spark environment. The research team improves recursive join with Semi-Naive algorithm using Intersection Bloom Filter [21,22,23] to remove redundant data that does not participate in the join operation to significantly reduce the costs involved.…”

Section: Introductionmentioning

confidence: 99%

Recursive Join Processing in Big Data Environment

Phan¹,

Trieu²,

Phan³

2021

JCC

Self Cite

View full text Add to dashboard Cite

In the era of information explosion, Big data is receiving increased attention as having important implications for growth, profitability, and survival of modern organizations. However, it also offers many challenges in the way data is processed and queried over time. A join operation is one of the most common operations appearing in many data queries. Specially, a recursive join is a join type used to query hierarchical data but it is more extremely complex and costly. The evaluation of the recursive join in MapReduce includes some iterations of two tasks of a join task and an incremental computation task. Those tasks are significantly expensive and reduce the performance of queries in large datasets because they generate plenty of intermediate data transmitting over the network. In this study, we thus propose a simple but efficient approach for Big recursive joins based on reducing by half the number of the required iterations in the Spark environment. This improvement leads to significantly reducing the number of the required tasks as well as the amount of the intermediate data generated and transferred over the network. Our experimental results show that an improved recursive join is more efficient and faster than a traditional one on large-scale datasets.

show abstract

Section: Introductionmentioning

confidence: 99%