A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Zhang, Feng; Zhou, Jingwei; Liu, Renyi; Du, Zhenhong; Ye, Xinyue

doi:10.3390/su8090926

Cited by 6 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the emergence of cloud computing, many studies use open source big data computing frameworks, such as Hadoop MapReduce and Apache Spark, to improve spatial join efficiency. Apart from the distributed spatial join algorithms included in the SASs of Table 1, in [33] the spatial join with Spark (SJS) was presented, and it used a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. Additionally, SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculationevaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access.…”

Section: Spatial Analytics Systemmentioning

confidence: 99%

“…Table 2 shows the syntheses of the implementations directly on Apache Spark of distributed algorithms with sophisticated processing techniques for other spatial queries, not using the previous SASs. Generic framework using clustering methods [28] In-memory partitioning and indexing system (SparkNN) SJQ [33] Spatial Join with Spark (SJS), uniform grid partitioning [34] Distributed join methods: Broadcast Join and Bin Join [35] Comparative study of common join algorithms in Spark TKSJQ [36] Uniform grid partitioning and improved plane-sweeping KNNJQ [37] Locality-Sensitive Hashing (LSH) algorithm in Spark MwSJQ [38] Multiway Spatial Join algorithm in Spark (MSJS), using cascaded pairwise join technique STSQ [39] Spark-based spatio-textual skyline query alg. (Multi-PSS) KCPQ, DJQ [40] SliceNBound (SnB), parent-child and common-merged strip partitioning and, plane-sweep technique [41] Strip-based partitioning and plane-sweep technique [42] Binary Space Partitioning (BSP).…”

Section: Spatial Analytics Systemmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Group K Nearest-Neighbor Spatial Query Processing in Apache Spark

Moutafis,

Mavrommatis,

Vassilakopoulos

et al. 2021

IJGI

View full text Add to dashboard Cite

Aiming at the problem of spatial query processing in distributed computing systems, the design and implementation of new distributed spatial query algorithms is a current challenge. Apache Spark is a memory-based framework suitable for real-time and batch processing. Spark-based systems allow users to work on distributed in-memory data, without worrying about the data distribution mechanism and fault-tolerance. Given two datasets of points (called Query and Training), the group K nearest-neighbor (GKNN) query retrieves (K) points of the Training with the smallest sum of distances to every point of the Query. This spatial query has been actively studied in centralized environments and several performance improving techniques and pruning heuristics have been also proposed, while, a distributed algorithm in Apache Hadoop was recently proposed by our team. Since, in general, Apache Hadoop exhibits lower performance than Spark, in this paper, we present the first distributed GKNN query algorithm in Apache Spark and compare it against the one in Apache Hadoop. This algorithm incorporates programming features and facilities that are specific to Apache Spark. Moreover, techniques that improve performance and are applicable in Apache Spark are also incorporated. The results of an extensive set of experiments with real-world spatial datasets are presented, demonstrating that our Apache Spark GKNN solution, with its improvements, is efficient and a clear winner in comparison to processing this query in Apache Hadoop.

show abstract

Section: Spatial Analytics Systemmentioning

confidence: 99%

Section: Spatial Analytics Systemmentioning

confidence: 99%

Efficient Group K Nearest-Neighbor Spatial Query Processing in Apache Spark

Moutafis,

Mavrommatis,

Vassilakopoulos

et al. 2021

IJGI

View full text Add to dashboard Cite

show abstract

“…Similar Spark-based spatial query systems also include Spark-GIS [22], Magellan [23], GeoTrellis [24] and LocationSpark [25]. In addition, some scholars studied the issues of high-performance spatial join queries based on Spark [26,27].…”

Section: Introductionmentioning

confidence: 99%

GeoSpark SQL: An Effective Framework Enabling Spatial Queries on Spark

Huang

Chen

Wan

et al. 2017

IJGI

View full text Add to dashboard Cite

Abstract:In the era of big data, Internet-based geospatial information services such as various LBS apps are deployed everywhere, followed by an increasing number of queries against the massive spatial data. As a result, the traditional relational spatial database (e.g., PostgreSQL with PostGIS and Oracle Spatial) cannot adapt well to the needs of large-scale spatial query processing. Spark is an emerging outstanding distributed computing framework in the Hadoop ecosystem. This paper aims to address the increasingly large-scale spatial query-processing requirement in the era of big data, and proposes an effective framework GeoSpark SQL, which enables spatial queries on Spark. On the one hand, GeoSpark SQL provides a convenient SQL interface; on the other hand, GeoSpark SQL achieves both efficient storage management and high-performance parallel computing through integrating Hive and Spark. In this study, the following key issues are discussed and addressed: (1) storage management methods under the GeoSpark SQL framework, (2) the spatial operator implementation approach in the Spark environment, and (3) spatial query optimization methods under Spark. Experimental evaluation is also performed and the results show that GeoSpark SQL is able to achieve real-time query processing. It should be noted that Spark is not a panacea. It is observed that the traditional spatial database PostGIS/PostgreSQL performs better than GeoSpark SQL in some query scenarios, especially for the spatial queries with high selectivity, such as the point query and the window query. In general, GeoSpark SQL performs better when dealing with compute-intensive spatial queries such as the kNN query and the spatial join query.

show abstract

“…Eldawy and Mokbel present a MapReduce-based spatial join method built in Spatial Hadoop, which is a MapReduce extension to Apache Hadoop that was designed specifically for spatial data [6]. In addition, with the development of efficient distributed memory computing platforms such as Spark, research has begun to focus on increasing the efficiency of spatial join queries with the help of Apache Spark [7][8][9][10].…”

Section: Introductionmentioning

confidence: 99%

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Zhao

et al. 2017

IJGI

Self Cite

View full text Add to dashboard Cite

Multiway spatial join plays an important role in GIS (Geographic Information Systems) and their applications. With the increase in spatial data volumes, the performance of multiway spatial join has encountered a computation bottleneck in the context of big data. Parallel or distributed computing platforms, such as MapReduce and Spark, are promising for resolving the intensive computing issue. Previous approaches have focused on developing single-threaded join algorithms as an optimizing and partition strategy for parallel computing. In this paper, we present an effective high-performance multiway spatial join algorithm with Spark (MSJS) to overcome the multiway spatial join bottleneck. MSJS handles the problem through cascaded pairwise join. Using the power of Spark, the formerly inefficient cascaded pairwise spatial join is transformed into a high-performance approach. Experiments using massive real-world data sets prove that MSJS outperforms existing parallel approaches of multiway spatial join that have been described in the literature.

show abstract

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Cited by 6 publications

References 37 publications

Efficient Group K Nearest-Neighbor Spatial Query Processing in Apache Spark

Efficient Group K Nearest-Neighbor Spatial Query Processing in Apache Spark

GeoSpark SQL: An Effective Framework Enabling Spatial Queries on Spark

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Contact Info

Product

Resources

About