Spatial coding-based approach for partitioning big spatial data in Hadoop

Abstract

Spatial data partitioning (SDP) plays a powerful role in distributed storage and parallel computing for spatial data. However, due to skew distribution of spatial data and varying volume of spatial vector objects, it leads to a significant challenge to ensure both optimal performance of spatial operation and data balance in the cluster. To tackle this problem, we proposed a spatial coding-based approach for partitioning big spatial data in Hadoop. This approach, firstly, compressed the whole big spatial data based on spatial coding matrix to create a sensing information set (SIS), including spatial code, size, count and other information. SIS was then employed to build spatial partitioning matrix, which was used to spilt all spatial objects into different partitions in the cluster finally. Based on our approach, the neighbouring spatial objects can be partitioned into the same block. At the same time, it also can minimize the data skew in Hadoop distributed file system (HDFS). The presented approach with a case study in this paper is compared against random sampling based partitioning, with three measurement standards, namely, the spatial index quality, data skew in HDFS, and range query performance. The experimental results show that our method based on spatial coding technique can improve the query performance of big spatial data, as well as the data balance in HDFS. We implemented and deployed this approach in Hadoop, and it is also able to support efficiently any other distributed big spatial data systems.

RDCRMG: A Raster Dataset Clean & Reconstitution Multi-Grid Architecture for Remote Sensing Monitoring of Vegetation Dryness

“…The application layer integrates the task request interface and result output interface. Furthermore, an RS data automatic pretreatment system (RSAPTS) [37][38][39][40] was constructed to support near real time GF-1 RS data updating, including automatic radiometric correction, orthorectification [41], cloud detection, geometric correction [42] and projection transformation [43], up to 1 December 2017, 12612 GF-1 images were processed and loaded into the VDMP for vegetation dryness monitoring.…”

Enabling the Big Earth Observation Data via Cloud Computing and DGGS: Opportunities and Challenges

“…For spatial data, cloud computing is unable to make the most of itself directly as it is designed ignoring characteristics of spatial dataset essentially [40]. Hence, spatial cloud computing (SCC) [16] For spatial data, cloud computing is unable to make the most of itself directly as it is designed ignoring characteristics of spatial dataset essentially [40]. Hence, spatial cloud computing (SCC) [16] is proposed.…”

“…Cloud computing provides computing technologies for the potential solution of transformation of big data's four Vs into the fifth V (value) [44]. The relevance and regionalization of spatial data hinder the optimal use of cloud computing technologies in GIS [40]. Based on the DGGS framework, the Earth is divided into multiple, continuous cells, and these discrete cells have the same geometry and the same area.…”

“…In terms of grid retrieval efficiency and performance, on the one hand, it needs to rely on the advantages of grid coding [90] and on the other hand, it needs to improve the query efficiency of data with the help of cloud computing [40]. Grid coding has been the hotspot of DGGS basic theory research [71,72,91].…”

Geospatial Information Processing Technologies

“…Some tools and studies use Hadoop as a black box for operations on data, such as GIS tools for Hadoop, a package composed of programming libraries and an add-on toolbox of ArcGIS desktop (ESRI 2018), and Hadoop-GIS, a scalable spatial data warehousing system (Aji et al 2013). Spatial Hadoop adds native support for spatial data by supporting a set of spatial index structures and developing spatial functions that interact directly with Hadoop base code (Yao et al 2017). Impala, a distributed SQL query engine for Hadoop, has also been extended for spatial data (Eldawy et al 2015).…”