Large Scale Analytics of Vector+Raster Big Spatial Data

Eldawy, Ahmed; Niu, Lyuye; Haynes, David; Su, Zhiba

doi:10.1145/3139958.3140042

Cited by 19 publications

(25 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [10], it was presented the scanline method, which is an algorithm to run the zonal statistics operation between a raster and a vector datasets without any previous transformation.…”

Section: Related Workmentioning

confidence: 99%

“…A linear region quadtree is used again to index the raster dataset, whereas the predictive nature of the operation requires a different index for the vector data, namely a TPR*-tree. Unfortunately, these works tackled only binary rasters, cells contain only black or white colours, and therefore they have a very limited real application.In [10], it was presented the scanline method, which is an algorithm to run the zonal statistics operation between a raster and a vector datasets without any previous transformation.Brisaboa et al[9] presented a framework to store and manage vector and compressed raster data, as well as an algorithm to solve a query that, given a vector and a raster dataset, returns the elements of the vector dataset overlapping regions of the raster dataset that fulfill a range constraint. For example, having a vector dataset representing the neighbourhoods of a city and a raster storing the amount of nitrogen oxides in the air, a query could be "return the neighbourhoods overlapping points December 30, 2019 3/36…”

mentioning

confidence: 99%

“…Other previous contributions deal with the implementation of query operators that are explicitly defined for querying datasets in different formats [7][8][9][10]. Some of them tackled the Join, or a close query, but in this case, these works suffer from limitations (data structures not functional enough, too restrictive join operations, size problems) that will be explained more in detail in the next section.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Efficient processing of raster and vector data

et al. 2020

View full text Add to dashboard Cite

In this work, we propose a framework to store and manage spatial data, which includes new efficient algorithms to perform operations accepting as input a raster dataset and a vector dataset. More concretely, we present algorithms for solving a spatial join between a raster and a vector dataset imposing a restriction on the values of the cells of the raster; and an algorithm for retrieving K objects of a vector dataset that overlap cells of a raster dataset, such that the K objects are those overlapping the highest (or lowest) cell values among all objects. The raster data is stored using a compact data structure, which can directly manipulate compressed data without the need for prior decompression. This leads to better running times and lower memory consumption. In our experimental evaluation comparing our solution to other baselines, we obtain the best space/time trade-offs.December 30, 2019 1/36 dataset, and then to use a raster algorithm to solve the query. This is the solution for the zonal statistics operation of Map Algebra in, at least, ArcGIS and GRASS [5, 6]. However, some previous research has addressed the problem using a joint approach. In [4], a single data model and language is proposed to represent and query both vector and raster data at the logical level. Even a Join operator is suggested, which allows combining, transparently and interchangeably, vector datasets, raster datasets, or both. As an example, the authors propose the query "return the coordinates of the trajectory of an aircraft when it was over a ground with altitude over 1,000". Unfortunately, no implementation details are given.Other previous contributions deal with the implementation of query operators that are explicitly defined for querying datasets in different formats [7][8][9][10]. Some of them tackled the Join, or a close query, but in this case, these works suffer from limitations (data structures not functional enough, too restrictive join operations, size problems) that will be explained more in detail in the next section.On the other hand, compression has been used traditionally with the aim of just reducing the size of the datasets in disk and during network transmissions. However, it has recently begun to be used as a way to obtain improvements in other dimensions, such as processing time or scalability [11]. In the last few years, several authors [12][13][14][15] have proposed the use of modern compact data structures [16] to represent raster datasets. Compact data structures use compression to reduce the size of the stored dataset, but with the novelty that the compressed data structure can be managed directly in compressed form, even in main memory. By saving main memory, we obtain a more scalable system, but at the same time, we take advantage of a better usage of the memory hierarchy, and thus obtain better running times. This strategy is sometimes called "in-memory" data management [17]. In addition, many compact data structures are equipped with an index that, in the same compressed space, speeds up the queries....

show abstract

“…In [10], it was presented the scanline method, which is an algorithm to run the zonal statistics operation between a raster and a vector datasets without any previous transformation.…”

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Efficient processing of raster and vector data

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The scan-line method [19] is the state-of-the-art algorithm for computing zonal statistics on a single machine. It runs in the following three steps.…”

Section: Single-machine Scanline Methodsmentioning

confidence: 99%

Distributed zonal statistics of big raster and vector data

Singla

Eldawy

2018

Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

Self Cite

View full text Add to dashboard Cite

Recent advancements in remote sensing technology have resulted in petabytes of data in raster format. This data is often processed in combination with high resolution vector data that represents, for example, city boundaries. One of the common operations that combine big raster and vector data is the zonal statistics which computes some statistics for each polygon in the vector dataset. This paper models the zonal statistics problem as a join problem and proposes a novel distributed system that can scale to petabytes of raster and vector data. The proposed method does not require any preprocessing or indexing which makes it perfect for ad-hoc queries that scientists usually want to run. We devise a theoretical cost model that proves the efficiency of our algorithm over the baseline method. Furthermore, we run an extensive experimental evaluation on large scale satellite data with up-to a trillion pixels, and big vector data with up-to hundreds of millions of edges, and we show that our method can perfectly scale to big data with up-to two orders of magnitude performance gain over Rasdaman and Google Earth Engine. This paper models the zonal statistics problem as a join problem as it needs to find the pixels in the raster layer that overlap the polygons in the vector layer. We explain, with a support of a theoretical analysis, that existing approaches are analogous to two common join algorithms, namely, index nested-loop join and hash join. This analogy highlights the main limitations of these two algorithms as the data size increases. Therefore, we propose a novel distributed algorithm, termed Raptor Zonal Statistics, which resembles the sort-merge join for big raster and vector data. We show

show abstract

“…Collectively the geospatial data available from several sources has grown into petabytes and increases by terabytes in size every day [24]. The increase in the sources of data and its acquisition have been exponential as compared to the development of processing systems which can process the data in real time.…”

Section: Big-geospatial Datamentioning

confidence: 99%

Multi-dimensional geospatial data mining in a distributed environment using MapReduce

Alkathiri

Abdul²,

Potdar³

2019

J Big Data

View full text Add to dashboard Cite

Data mining and machine learning techniques for processing raster data consider a single spectral band of data at a time. The individual results are combined to obtain the final output. The essence of related multi-spectral information is lost when the bands are considered independently. The proposed platform is based on Apache Hadoop ecosystem and supports performing analysis on large amounts of multispectral raster data using MapReduce. A novel technique of transforming the spectral space to the geometrical space is also proposed. The technique allows to consider multiple bands coherently. The results of clustering 10 6 pixels for multiband imagery with widely used GIS software have been tested and other machine learning methods are planned to be incorporated in the platform. The platform is scalable to support tens of spectral bands. The results from our platform were found to be better and are also available faster due to application of distributed processing.

show abstract

Large Scale Analytics of Vector+Raster Big Spatial Data

Cited by 19 publications

References 11 publications

Efficient processing of raster and vector data

Efficient processing of raster and vector data

Distributed zonal statistics of big raster and vector data

Multi-dimensional geospatial data mining in a distributed environment using MapReduce

Contact Info

Product

Resources

About