A spatiotemporal indexing approach for efficient processing of big array-based climate data with MapReduce

Li, Zhenlong; Hu, Fei; Schnase, John L.; Duffy, Daniel Q.; Lee, Tsengdar; Bowen, M. K.; Yang, Chaowei

doi:10.1080/13658816.2015.1131830

Cited by 56 publications

(35 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Krishnan et al investigated the use of MapReduce to generate DEM by gridding the LIDAR data [22]. Li et al utilized Hadoop MapReduce to enable penalization of big climate data processing [11,13]. Besides these problem-specific approaches focusing on solving specific problems with Hadoop, tools have also been developed to handle general geospatial data processing tasks and are being adopted in GIScience communities.…”

Section: Hadoop For Geospatial Data Processingmentioning

confidence: 99%

“…Hadoop, a distributed computing platform leveraging commodity computers, is gaining increasing popularity in geoscience communities, as reviewed in Section 2. While a lot of effort was put into investigating how to adapt Hadoop for processing big geospatial data (e.g., [9][10][11][12][13]), how to efficiently handle different geoprocessing workload by dynamically adjusting the amount of computing resources (number of nodes of a Hadoop cluster) was barely explored. The ability to dynamically adjust the computing resources is important because the processing workload of operational geospatial applications is rather dynamic than static [14]; for example, the data processing workload for an emergency response system (such as for wildfires, tsunami, and earthquakes) peaks during the emergency event, which requires adequate computing power to respond promptly [14,15].…”

Section: Introductionmentioning

confidence: 99%

“…Such a cloud-enabled, auto-scalable computing cluster offers a powerful tool to process big geoscience data with optimized performance and reduced resource consumption. While DEM interpolation is used as an example, the proposed framework can be extended to handle other geoprocessing applications that run on Hadoop, such as the climate data analytical services powered by Hadoop and cloud computing [7,11]. In addition, the predictive scaling algorithm sheds lights on automatically scaling cloud computing resources for other data processing platforms beyond Hadoop.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

Yang

Liu

et al. 2016

IJGI

Self Cite

View full text Add to dashboard Cite

Efficient processing of big geospatial data is crucial for tackling global and regional challenges such as climate change and natural disasters, but it is challenging not only due to the massive data volume but also due to the intrinsic complexity and high dimensions of the geospatial datasets. While traditional computing infrastructure does not scale well with the rapidly increasing data volume, Hadoop has attracted increasing attention in geoscience communities for handling big geospatial data. Recently, many studies were carried out to investigate adopting Hadoop for processing big geospatial data, but how to adjust the computing resources to efficiently handle the dynamic geoprocessing workload was barely explored. To bridge this gap, we propose a novel framework to automatically scale the Hadoop cluster in the cloud environment to allocate the right amount of computing resources based on the dynamic geoprocessing workload. The framework and auto-scaling algorithms are introduced, and a prototype system was developed to demonstrate the feasibility and efficiency of the proposed scaling mechanism using Digital Elevation Model (DEM) interpolation as an example. Experimental results show that this auto-scaling framework could (1) significantly reduce the computing resource utilization (by 80% in our example) while delivering similar performance as a full-powered cluster; and (2) effectively handle the spike processing workload by automatically increasing the computing resources to ensure the processing is finished within an acceptable time. Such an auto-scaling approach provides a valuable reference to optimize the performance of geospatial applications to address data-and computational-intensity challenges in GIScience in a more cost-efficient manner.

show abstract

Section: Hadoop For Geospatial Data Processingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

Yang

Liu

et al. 2016

IJGI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Hadoop has achieved an unprecedented success in implementing many real-world distributed tasks. There are some researches focus on spatial data storage and query under Hadoop framework, such as SpatialHadoop (Eldawy & Mokbel, 2015), Hadoop-GIS (Aji et al, 2013), SciHive (Geng, Huang, Zhu, Ruan, & Yang, 2013), GeoMesa (Fox, Eichelberger, Hughes, & Lyon, 2013), Terry Fly (Cary, Yesha, Adjouadi, & Rishe, 2010), and GeoBase (Li, Hu, Schnase, & Duffy, 2016), and so on. However, the coupling among MapReduce model and HDFS is high.…”

Section: Parallel and Distributed Spatial Data Indexingmentioning

confidence: 99%

A Novel Approach of Indexing and Retrieving Spatial Polygons for Efficient Spatial Region Queries

Zhao

Wang

et al. 2017

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

ABSTRACT:Spatial region queries are more and more widely used in web-based applications. Mechanisms to provide efficient query processing over geospatial data are essential. However, due to the massive geospatial data volume, heavy geometric computation, and high access concurrency, it is difficult to get response in real time. Spatial indexes are usually used in this situation. In this paper, based on k-d tree, we introduce a distributed KD-Tree (DKD-Tree) suitbable for polygon data, and a two-step query algorithm. The spatial index construction is recursive and iterative, and the query is an in memory process. Both the index and query methods can be processed in parallel, and are implemented based on HDFS, Spark and Redis. Experiments on a large volume of Remote Sensing images metadata have been carried out, and the advantages of our method are investigated by comparing with spatial region queries executed on PostgreSQL and PostGIS. Results show that our approach not only greatly improves the efficiency of spatial region query, but also has good scalability, Moreover, the two-step spatial range query algorithm can also save cluster resources to support a large number of concurrent queries. Therefore, this method is very useful when building large geographic information systems.

show abstract

“…• Based on the good scalability, large-scale concurrent processing capability and MapReduce parallel model of NoSQL databases, some preliminary research has been made toward exploring spatial data distributed storage and processing [20][21][22][23][24].…”

Section: Not Only Sql (Nosql)-enabled Big Data Managementmentioning

confidence: 99%

An Effective NoSQL-Based Vector Map Tile Management Approach

Wan

Huang

Xia

2016

IJGI

View full text Add to dashboard Cite

Within a digital map service environment, the rapid growth of Spatial Big-Data is driving new requirements for effective mechanisms for massive online vector map tile processing. The emergence of Not Only SQL (NoSQL) databases has resulted in a new data storage and management model for scalable spatial data deployments and fast tracking. They better suit the scenario of high-volume, low-latency network map services than traditional standalone high-performance computer (HPC) or relational databases. In this paper, we propose a flexible storage framework that provides feasible methods for tiled map data parallel clipping and retrieval operations within a distributed NoSQL database environment. We illustrate the parallel vector tile generation and querying algorithms with the MapReduce programming model. Three different processing approaches, including local caching, distributed file storage, and the NoSQL-based method, are compared by analyzing the concurrent load and calculation time. An online geological vector tile map service prototype was developed to embed our processing framework in the China Geological Survey Information Grid. Experimental results show that our NoSQL-based parallel tile management framework can support applications that process huge volumes of vector tile data and improve performance of the tiled map service.

show abstract

A spatiotemporal indexing approach for efficient processing of big array-based climate data with MapReduce

Cited by 56 publications

References 14 publications

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

A Novel Approach of Indexing and Retrieving Spatial Polygons for Efficient Spatial Region Queries

An Effective NoSQL-Based Vector Map Tile Management Approach

Contact Info

Product

Resources

About