Skew-resistant parallel in-memory spatial join

Ray, Suprio; Simion, Bogdan; Brown, Angela Demke; Johnson, Ryan

doi:10.1145/2618243.2618262

Cited by 26 publications

(12 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Distributed and parallel techniques. There are also recent works on addressing irregular polygon range queries using distributed partitioning techniques [14,24,25,28] and parallel GPUbased techniques [1,33,34]. These techniques mostly rely on partitioning data across a cluster of machines or GPU cores so that one query is partitioned along with data partitioning, and then the query executes on multiple nodes/cores that have relevant data.…”

Section: Related Workmentioning

confidence: 99%

Scalable Spatial GroupBy Aggregations Over Complex Polygons

Abdelhafeez

Magdy

Tsotras

2020

Proceedings of the 28th International Conference on Advances in Geographic Information Systems

View full text Add to dashboard Cite

This paper studies a spatial group-by query over complex polygons. Groups are selected from a set of non-overlapping complex polygons, typically in the order of thousands, while the input is a large-scale dataset that contains hundreds of millions or even billions of spatial points. Given a set of spatial points and a set of polygons, the spatial group-by query returns the number of points that lie within boundaries of each polygon. This problem is challenging because real polygons (like counties, cities, postal codes, voting regions, etc.) are described by very complex boundaries. We propose a highly-parallelized query processing framework to efficiently compute the spatial group-by query. Our experimental evaluation with real data and queries has shown significant superiority over all existing techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

Scalable Spatial GroupBy Aggregations Over Complex Polygons

Abdelhafeez

Magdy

Tsotras

2020

Proceedings of the 28th International Conference on Advances in Geographic Information Systems

View full text Add to dashboard Cite

show abstract

“…Many classical methods have been proposed to reduce the candidate set by a filter step via sorting or indexing one or more datasets [3,14,21]. Some technologies have been developed to add a compute-intensive refinement step to further improve the performance [23].…”

Section: Background and Related Workmentioning

confidence: 99%

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Zhao

et al. 2017

IJGI

View full text Add to dashboard Cite

Multiway spatial join plays an important role in GIS (Geographic Information Systems) and their applications. With the increase in spatial data volumes, the performance of multiway spatial join has encountered a computation bottleneck in the context of big data. Parallel or distributed computing platforms, such as MapReduce and Spark, are promising for resolving the intensive computing issue. Previous approaches have focused on developing single-threaded join algorithms as an optimizing and partition strategy for parallel computing. In this paper, we present an effective high-performance multiway spatial join algorithm with Spark (MSJS) to overcome the multiway spatial join bottleneck. MSJS handles the problem through cascaded pairwise join. Using the power of Spark, the formerly inefficient cascaded pairwise spatial join is transformed into a high-performance approach. Experiments using massive real-world data sets prove that MSJS outperforms existing parallel approaches of multiway spatial join that have been described in the literature.

show abstract

“…However, the partition strategy in Niharika considers memory only; in other words, the recursive tiling is only based on the numbers of objects, which may cause uneven spatial join processing in each partition, especially when the number of objects of the two datasets are similar in the same partition. Skew-resistant Parallel IN-memOry spatial Join Architecture (SPINOJA) [30] focuses on the performance of the refinement stage by considering that the efficiency bottleneck is the processing skew caused by the uneven work metrics, including object size and point density. However, in this approach, objects ore decomposed by clipping against the tile boundaries in order to reduce processing skew on large objects, which could require additional storage space or calculation costs and increase the number of objects.…”

Section: Spatial Join Querymentioning

confidence: 99%

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

et al. 2016

View full text Add to dashboard Cite

Sustainability research faces many challenges as respective environmental, urban and regional contexts are experiencing rapid changes at an unprecedented spatial granularity level, which involves growing massive data and the need for spatial relationship detection at a faster pace. Spatial join is a fundamental method for making data more informative with respect to spatial relations. The dramatic growth of data volumes has led to increased focus on high-performance large-scale spatial join. In this paper, we present Spatial Join with Spark (SJS), a proposed high-performance algorithm, that uses a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculation-evaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access. We compare four in-memory spatial join algorithms in SJS for further performance improvement. Based on extensive experiments with real-world data, we conclude that SJS outperforms the Spark and MapReduce implementations of earlier spatial join approaches. This study demonstrates that it is promising to leverage high-performance computing for large-scale spatial join analysis. The availability of large-sized geo-referenced datasets along with the high-performance computing technology can raise great opportunities for sustainability research on whether and how these new trends in data and technology can be utilized to help detect the associated trends and patterns in the human-environment dynamics.

show abstract

Skew-resistant parallel in-memory spatial join

Cited by 26 publications

References 18 publications

Scalable Spatial GroupBy Aggregations Over Complex Polygons

Scalable Spatial GroupBy Aggregations Over Complex Polygons

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Contact Info

Product

Resources

About