GeoSpark SQL: An Effective Framework Enabling Spatial Queries on Spark

Abstract

Abstract:In the era of big data, Internet-based geospatial information services such as various LBS apps are deployed everywhere, followed by an increasing number of queries against the massive spatial data. As a result, the traditional relational spatial database (e.g., PostgreSQL with PostGIS and Oracle Spatial) cannot adapt well to the needs of large-scale spatial query processing. Spark is an emerging outstanding distributed computing framework in the Hadoop ecosystem. This paper aims to address the increasingly large-scale spatial query-processing requirement in the era of big data, and proposes an effective framework GeoSpark SQL, which enables spatial queries on Spark. On the one hand, GeoSpark SQL provides a convenient SQL interface; on the other hand, GeoSpark SQL achieves both efficient storage management and high-performance parallel computing through integrating Hive and Spark. In this study, the following key issues are discussed and addressed: (1) storage management methods under the GeoSpark SQL framework, (2) the spatial operator implementation approach in the Spark environment, and (3) spatial query optimization methods under Spark. Experimental evaluation is also performed and the results show that GeoSpark SQL is able to achieve real-time query processing. It should be noted that Spark is not a panacea. It is observed that the traditional spatial database PostGIS/PostgreSQL performs better than GeoSpark SQL in some query scenarios, especially for the spatial queries with high selectivity, such as the point query and the window query. In general, GeoSpark SQL performs better when dealing with compute-intensive spatial queries such as the kNN query and the spatial join query.

Trajectory Clustering and k-NN for Robust Privacy Preserving k-NN Query Processing in GeoSpark

“…GeoSpark [32] is an extension of the Apache Spark core and provides additional tools to manage geospatial data, e.g., geospatial datatypes, indexes and operations. The architecture of GeoSpark, as introduced in Figure 3, consists of the following three layers:…”

BiGeo: A Foundational PaaS Framework for Efficient Storage, Visualization, Management, Analysis, Service, and Migration of Geospatial Big Data—A Case Study of Sichuan Province, China

“…GeoSpark has a range of creative spatially flexible distributed datasets. With the help of these datasets, users can efficiently load, process, and analyze large-scale spatial data [35,59,60].…”

Spatiotemporal Aspects of Big Data

“…In future, the social media data is also going to be integrated with this classification method. According to [61], the authors state that geospatial information is increasing along with the increase of big data. Spark framework is developed for distributed computing framework in Hadoop ecosystems.…”

“…[24], [30], [57], [61], [71], [73] Apache Flink [26], [58], [67], [73] Processing Table II and Table III. The focus of study is to portrait work done in the aspect of spatiotemporal big data on different frameworks.…”