Lessons Learnt from the Development of GIS Application on Azure Cloud Platform

Abstract

Spatial overlay processing is a widely used computeintensive GIS application that involves aggregation of two or more layers of maps to facilitate intelligent querying on the collocated output data. When large GIS data sets are represented in polygonal (vector) form, spatial analysis runs for extended periods of time, which is undesirable for time-sensitive applications such as emergency response. We have, for the first time, created an open-architecture-based system named Crayons for Azure cloud platform using state-of-the-art techniques. During the course of development of Crayons system, we faced numerous challenges and gained invaluable insights into Azure cloud platform, which are presented in detail in this paper. The challenges range from limitations of cloud storage and computational services to the choices of tools and technologies used for high performance computing (HPC) application design. We report our findings to provide concrete guidelines to an eScience developer for 1) choice of persistent data storage mechanism, 2) data structure representation, 3) communication and synchronization among nodes, 4) building robust failsafe applications, and 5) optimal cost-effective utilization of resources. Our insights into each challenge faced, the solution to overcome it, and the discussion on the lessons learnt from each challenge can be of help to eScience developers starting application development on Azure and possibly other cloud platforms.

MapReduce Algorithms for GIS Polygonal Overlay Processing

“…We presented a parallelization scheme for polygonal overlay based on Message Passing Interface (MPI) in [1]. In [15] and [16], we discuss static and dynamic load balancing strategies for parallelizing map overlay on Azure cloud platform. But these parallelization schemes are not applicaple to Hadoop-based MapReduce system since the latter lacks explicit message passing as in MPI framework (point to point send/receive) or the queues as in Azure for communication.…”

Bivariate, cluster, and suitability analysis of NoSQL solutions for big graph applications

“…Geo-data is another typically large dataset that needs to be analyzed to gain useful insights and make predictions that are crucial for mission critical applications and projects. The applications associated with this domain span across data representation [223], semantic data management [212], big geospatial raster data management [215] and geospatial/GIS applications [224,264].…”

GPGPU-accelerated interesting interval discovery and other computations on GeoSpatial datasets

“…Parallel Polygon Overlay System: Our GIS system employing the parallel R-tree can process about 200K polygons within a few seconds with 70-fold speedup on a (12-node Linux cluster that previously took tens of minutes [25,12,1,3,2]). Thus, it has the potential for bringing a practical overlay tool to the Geo Scientists.…”