An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Du, Zhenhong; Zhao, Xianwei; Ye, Xinyue; Zhou, Jingwei; Zhang, Feng; Liu, Renyi

doi:10.3390/ijgi6040096

Cited by 21 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In overlay analysis [24], the utilization of a uniform grid partition method accelerates the overlay process. Likewise, a grid partition strategy has also been applied in spatial join processing [25]. As mentioned in these two articles, the Spark environment provides positive results.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Approach to Accelerating Dissolved Vector Buffer Generation in Distributed In-Memory Cluster Architecture

Shen

Chen

et al. 2018

IJGI

View full text Add to dashboard Cite

Abstract:The buffer generation algorithm is a fundamental function in GIS, identifying areas of a given distance surrounding geographic features. Past research largely focused on buffer generation algorithms generated in a stand-alone environment. Moreover, dissolved buffer generation is dataand computing-intensive. In this scenario, the improvement in the stand-alone environment is limited when considering large-scale mass vector data. Nevertheless, recent parallel dissolved vector buffer algorithms suffer from scalability problems, leaving room for further optimization. At present, the prevailing in-memory cluster-computing framework-Spark-provides promising efficiency for computing-intensive analysis; however, it has seldom been researched for buffer analysis. On this basis, we propose a cluster-computing-oriented parallel dissolved vector buffer generating algorithm, called the HPBM, that contains a Hilbert-space-filling-curve-based data partition method, a data skew and cross-boundary objects processing strategy, and a depth-given tree-like merging method. Experiments are conducted in both stand-alone and cluster environments using real-world vector data that include points and roads. Compared with some existing parallel buffer algorithms, as well as various popular GIS software, the HPBM achieves a performance gain of more than 50%.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Unfortunately, these extended frameworks do not yet support buffer processing. However, we observe that Spark has already been used in spatial overlay analysis [24] and spatial multi-way join calculation [25], suggesting its potential application to buffer analysis.…”

Section: Input Buffers With Intact Boundariesmentioning

confidence: 99%

Approach to Accelerating Dissolved Vector Buffer Generation in Distributed In-Memory Cluster Architecture

Shen

Chen

et al. 2018

IJGI

View full text Add to dashboard Cite

show abstract

“…Existing studies also demonstrate that Spark has the potential to become an excellent spatial data management and computing framework [18][19][20][21][22][23][24][25][26][27]. Therefore, we expect to explore a Spark-based spatial query implementation framework i.e., GeoSpark SQL, to provide a convenient SQL query interface, and to achieve high-performance computing at the same time.…”

Section: Design Issuesmentioning

confidence: 99%

“…Similar Spark-based spatial query systems also include Spark-GIS [22], Magellan [23], GeoTrellis [24] and LocationSpark [25]. In addition, some scholars studied the issues of high-performance spatial join queries based on Spark [26,27].…”

Section: Introductionmentioning

confidence: 99%

GeoSpark SQL: An Effective Framework Enabling Spatial Queries on Spark

Huang

Chen

Wan

et al. 2017

IJGI

View full text Add to dashboard Cite

Abstract:In the era of big data, Internet-based geospatial information services such as various LBS apps are deployed everywhere, followed by an increasing number of queries against the massive spatial data. As a result, the traditional relational spatial database (e.g., PostgreSQL with PostGIS and Oracle Spatial) cannot adapt well to the needs of large-scale spatial query processing. Spark is an emerging outstanding distributed computing framework in the Hadoop ecosystem. This paper aims to address the increasingly large-scale spatial query-processing requirement in the era of big data, and proposes an effective framework GeoSpark SQL, which enables spatial queries on Spark. On the one hand, GeoSpark SQL provides a convenient SQL interface; on the other hand, GeoSpark SQL achieves both efficient storage management and high-performance parallel computing through integrating Hive and Spark. In this study, the following key issues are discussed and addressed: (1) storage management methods under the GeoSpark SQL framework, (2) the spatial operator implementation approach in the Spark environment, and (3) spatial query optimization methods under Spark. Experimental evaluation is also performed and the results show that GeoSpark SQL is able to achieve real-time query processing. It should be noted that Spark is not a panacea. It is observed that the traditional spatial database PostGIS/PostgreSQL performs better than GeoSpark SQL in some query scenarios, especially for the spatial queries with high selectivity, such as the point query and the window query. In general, GeoSpark SQL performs better when dealing with compute-intensive spatial queries such as the kNN query and the spatial join query.

show abstract

“…Um exemplo da adoção da presente diretriz pode ser encontrado no desenvolvimento de algoritmos destinados a otimizar consultas de junção espacial baseadas em distância (GARCÍA-GARCÍA et al, 2017). Outro exemplo refere-se ao algoritmo descrito em Du et al (2017), o qual utiliza o Spark para processar junções espaciais multidirecionais de forma eficiente. Pode-se citar ainda como exemplo um algoritmo para otimizar o desempenho de junções espaciais baseadas na intersecção de polilinhas em clusters acelerados por Graphics Processing Units (GPUs) (YOU; ZHANG; GRUENWALD, 2016).…”

Section: Diretriz 5: Foco Em Eficiênciaunclassified

Analisando Sistemas Analíticos Espaciais Baseados em Hadoop e Spark: Uma Perspectiva de Usuário

Castro¹

View full text Add to dashboard Cite

Sistemas Analíticos Espaciais (SAEs) representam uma nova tecnologia capaz de gerenciar um grande volume de dados espaciais por meio da utilização de frameworks de processamento paralelo e distribuído de dados, tais como o Hadoop e o Spark. Um número crescente de SAEs tem sido proposto na literatura, fato que evidencia a necessidade de se realizar análises comparativas entre esses sistemas. No entanto, as comparações disponíveis no estado da arte fornecem apenas uma visão centrada no desempenho dos SAEs. Ou seja, no melhor do conhecimento do autor do presente trabalho, não existem abordagens na literatura que realizem comparações entre SAEs com base em uma visão centrada no usuário, ou seja, comparações que visam ajudar os usuários a entender como as características dos SAEs são úteis para atender aos requisitos específicos de suas aplicações espaciais. No presente trabalho, essa lacuna na literatura é preenchida. Uma comparação dos seguintes SAEs baseados em Hadoop e Spark é fornecida, utilizando como base a perspectiva de seus usuários: Hadoop-GIS, SpatialHadoop, SpatialSpark, GeoSpark, GeoMesa Spark, SIMBA, LocationSpark, STARK, Magellan, SparkGIS e Elcano. Essa comparação é realizada de acordo com um amplo conjunto de critérios relacionados às características gerais desses sistemas, aos aspectos de manipulação de dados espaciais e aos aspectos inerentes ao ambiente distribuído. Com base nessa comparação, um conjunto de diretrizes é introduzido a fim de ajudar os usuários no processo de escolha de um SAE apropriado. Dois estudos de caso baseados em aplicações do mundo real também são descritos para ilustrar a aplicabilidade dessas diretrizes. Por fim, também são realizadas discussões sobre tendências cronológicas relacionadas aos SAEs e sobre as limitações que esses sistemas devem suprir a fim de aprimorar a experiência do usuário.

show abstract

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Cited by 21 publications

References 32 publications

Approach to Accelerating Dissolved Vector Buffer Generation in Distributed In-Memory Cluster Architecture

Approach to Accelerating Dissolved Vector Buffer Generation in Distributed In-Memory Cluster Architecture

GeoSpark SQL: An Effective Framework Enabling Spatial Queries on Spark

Analisando Sistemas Analíticos Espaciais Baseados em Hadoop e Spark: Uma Perspectiva de Usuário

Contact Info

Product

Resources

About