High performance analysis of big spatial data

Haynes, David; Ray, Suprio; Manson, Steven M.; Soni, Ankit Narendrakumar

doi:10.1109/bigdata.2015.7363974

Cited by 19 publications

(18 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhou et al (2015) The experiment results show that this approach is 2x faster than the state-of-the-art. Haynes et al (2015) introduces Terra Populus that acts as the bridge between big data sources and researchers. Researchers are provided with convenient web applications that allow them to access, analyze, and tabulate different datasets under a common platform.…”

Section: Analytics and Conclusionmentioning

confidence: 99%

A Systematic Review of SQL-on-Hadoop by Using Compact Data Formats

Plase¹

2017

BJMC

View full text Add to dashboard Cite

There are huge volumes of raw data generated every day. The question is how to store these data in order to provide faster data access. The research direction in Big Data projects using Hadoop Technology, MapReduce kind of framework and compact data formats shows that two data formats (Avro and Parquet) support schema evolution and compression in order to utilize less storage space. In this paper, a systematic review of SQL-on-Hadoop by using Avro and Parquet has been performed over the past seven years (2010-2016) using publications of conference proceedings and journals of IEEEXplore, ACM Digital Library, ScienceDirect. With the help of search strategy followed, 152 research papers have been identified out of which 27 (from year 2013-2016) have been analyzed deeply as relevant papers. At the end, the conclusion has been made that direct comparison by compactness and fastness between Avro and Parquet do not exist in analyzed scientific articles.

show abstract

Section: Analytics and Conclusionmentioning

confidence: 99%

A Systematic Review of SQL-on-Hadoop by Using Compact Data Formats

Plase¹

2017

BJMC

View full text Add to dashboard Cite

show abstract

“…In particular, most approaches to parallelization are limited in scope or provide extensions to existing frameworks such as MapReduce and column store databases. While this work is very promising, no existing systems are robust or wide-ranging enough for GIScience production environments like TerraPop (Haynes et al 2015). …”

Section: Spatial High-performance Computingmentioning

confidence: 99%

“…We chose this approach based on evidence that parallel relational databases like PostgreSQL can perform significantly better than MapReduce systems (Pavlo et al 2009). Our work to date has significantly improved performance in analyzing vector datasets, offering near linear speedup when adding nodes by sharding spatial queries across a cluster of machines where a PostgreSQL database instance is run on each node for simple topographical operations such as determining whether a polygon intersects a line or other polygon (Haynes et al 2015; Ray et al 2014). This work thereby addresses fundamental research needs in spatial high performance computing (Vo, Aji, and Wang 2014).…”

Section: Spatial High-performance Computingmentioning

confidence: 99%

Deserts in the Deluge: TerraPopulus and Big Human-Environment Data

Manson

Kugler²,

Haynes³

2016

GIScience

Self Cite

View full text Add to dashboard Cite

Terra Populus, or TerraPop, is a cyberinfrastructure project that integrates, preserves, and disseminates massive data collections describing characteristics of the human population and environment over the last six decades. TerraPop has made a number of GIScience advances in the handling of big spatial data to make information interoperable between formats and across scientific communities. In this paper, we describe challenges of these data, or ‘deserts in the deluge’ of data, that are common to spatial big data more broadly, and explore computational solutions specific to microdata, raster, and vector data models.

show abstract

“…PostgreSQL does not natively support parallel queries for analyzing many kinds of big spatial data, and parallelization (essentially breaking a big problem into many small ones handled by separate computing cores) is the primary way in which big data is handled by most computation platforms. In our previous work we extended upon the existing PostgreSQL platform (Haynes, Ray, Manson, & Soni, ; Ray, Simion, Brown, & Johnson, ). Haynes et al () discussed a prototype shared‐nothing parallel spatial database, called SpatialStado.…”

Section: Introductionmentioning

confidence: 99%

“…In our previous work we extended upon the existing PostgreSQL platform (Haynes, Ray, Manson, & Soni, ; Ray, Simion, Brown, & Johnson, ). Haynes et al () discussed a prototype shared‐nothing parallel spatial database, called SpatialStado. SpatialStado acts as a coordinating node allowing it to leverage multiple PostgreSQL instances but currently has not implemented the raster data type.…”

Section: Introductionmentioning

confidence: 99%

Terra Populus’ architecture for integrated big geospatial services

Haynes

Manson

Shook

2017

Transactions in GIS

Self Cite

View full text Add to dashboard Cite

Big geospatial data is an emerging sub-area of geographic information science, big data, and cyberinfrastructure. Big geospatial data poses two unique challenges to these and other cognate disciplines. First, raster and vector data structures and analyses have developed on largely separate paths for the last twenty years and this creates an impediment to researchers utilizing big data platforms that do not promote the integration for these classes. Second, big spatial data repositories have yet to be integrated with big data computation platforms in ways that allow researchers to spatio-temporally analyze big geospatial datasets. IPUMS-Terra, a National Science Foundation cyberInfrastructure project, begins to address these challenges. IPUMS-Terra is a spatial data infrastructure project that provides a unified framework for accessing, analyzing, and transforming big heterogeneous spatio-temporal data, and is part of the IPUMS (Integrated Public Use Microdata Series) data infrastructure. It supports big geospatial data analysis and provides integrated big geospatial services to its users. As IPUMS-Terra’s data volume grows, we seek to integrate geospatial platforms that will scale geospatial analyses and address current bottlenecks within our system. However, our work shows that there are still unresolved challenges for big geospatial analysis. The most pertinent is that there is a lack of a unified framework for conducting scalable integrated vector and raster data analysis. We conducted a comparative analysis between PostgreSQL with PostGIS and SciDB and concluded that SciDB is the superior platform for scalable raster zonal analyses.

show abstract

High performance analysis of big spatial data

Cited by 19 publications

References 5 publications

A Systematic Review of SQL-on-Hadoop by Using Compact Data Formats

A Systematic Review of SQL-on-Hadoop by Using Compact Data Formats

Deserts in the Deluge: TerraPopulus and Big Human-Environment Data

Terra Populus’ architecture for integrated big geospatial services

Contact Info

Product

Resources

About