A Comparison of Distributed Spatial Data Management Systems for Processing Distance Join Queries

Abstract

Abstract. Due to the ubiquitous use of spatial data applications and the large amounts of spatial data that these applications generate, the processing of large-scale distance joins in distributed systems is becoming increasingly popular. Two of the most studied distance join queries are the K Closest Pair Query (KCPQ) and the ε Distance Join Query (εDJQ). The KCPQ finds the K closest pairs of points from two datasets and the εDJQ finds all the possible pairs of points from two datasets, that are within a distance threshold ε of each other. Distributed clusterbased computing systems can be classified in Hadoop-based and Sparkbased systems. Based on this classification, in this paper, we compare two of the most current and leading distributed spatial data management systems, namely SpatialHadoop and LocationSpark, by evaluating the performance of existing and newly proposed parallel and distributed distance join query algorithms in different situations with big real-world datasets. As a general conclusion, while SpatialHadoop is more mature and robust system, LocationSpark is the winner with respect to the total execution time.

Analyzing spatial analytics systems based on Hadoop and Spark: A user perspective

“…The latest systems have been Spark‐based, that is, SpatialSpark, GeoSpark, GeoMesa Spark, Simba, LocationSpark, STARK, Magellan, SparkGIS, and Elcano. Furthermore, the work presented in Reference 34 introduces algorithms for optimizing the distance join queries and implement them using SpatialHadoop and LocationSpark. Differently from our work, approaches in this first group only briefly and technically summarize system by system.…”

“…Although the frameworks described in this section are efficient for processing conventional data, such as numeric and alphanumeric data types, they do not offer native support for processing spatial data efficiently. For instance, they do not provide indexing mechanisms that are capable of quickly retrieving groups of spatial objects that satisfy a given topological predicate, preventing the development of more efficient query processing algorithms 34 . SASs have emerged to overcome this limitation.…”

Analisando Sistemas Analíticos Espaciais Baseados em Hadoop e Spark: Uma Perspectiva de Usuário

“…Sistemas Analíticos Espaciais (SAEs) surgiram como uma possível solução para essa demanda. Esses sistemas fornecem funcionalidades especializadas para processar e indexar grandes volumes de dados espaciais por meio da utilização de frameworks de processamento paralelo e distribuído de dados (GARCÍA-GARCÍA et al, 2017). Na literatura, os frameworks Hadoop (APACHE, 2019a) e Spark (APACHE, 2019d) se destacam, uma vez que a maioria dos SAEs existentes são baseados nos mesmos (PANDEY et al, 2018).…”

“…Além disso, somente SAEs capazes de representar dados espaciais por meio do modelo vetorial foram selecionados. Ademais, propostas existentes no estado da arte que consistem em otimizações muito específicas de um único tipo de operação espacial não foram consideradas na presente análise comparativa, tais como otimizações específicas para o processamento de consultas baseadas em distância (GARCÍA-GARCÍA et al, 2017).…”

“…Um exemplo da adoção da presente diretriz pode ser encontrado no desenvolvimento de algoritmos destinados a otimizar consultas de junção espacial baseadas em distância (GARCÍA-GARCÍA et al, 2017). Outro exemplo refere-se ao algoritmo descrito em Du et al (2017), o qual utiliza o Spark para processar junções espaciais multidirecionais de forma eficiente.…”