The Raptor Join Operator for Processing Big Raster + Vector Data

Singla, Samriddhi; Eldawy, Ahmed; Diao, Tina; Mukhopadhyay, Ayan; Scudiero, Elia

doi:10.1145/3474717.3483971

Cited by 11 publications

(4 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To tackle such issues, we designed WildfireDB, a collection of over 17 million data points that capture fire spread in continental United States in the last decade [10]. Our data generation, to a large extent, is based on work on largescale vector and raster analysis [15], and uses a principled algorithmic approach to merge large-scale raster and vector data. WildfireDB can be used to model the spread of wildfires and inform response strategies [16].…”

Section: A Challenges In Data Processingmentioning

confidence: 99%

Designing Decision Support Systems for Emergency Response: Challenges and Opportunities

Pettet¹,

Baxter²,

Vazirizade³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Designing effective emergency response management (ERM) systems to respond to incidents such as road accidents is a major problem faced by communities. In addition to responding to frequent incidents each day (about 240 million emergency medical services calls and over 5 million road accidents in the US each year), these systems also support response during natural hazards. Recently, there has been a consistent interest in building decision support and optimization tools that can help emergency responders provide more efficient and effective response. This includes a number of principled subsystems that implement early incident detection, incident likelihood forecasting and strategic resource allocation and dispatch policies. In this paper, we highlight the key challenges and provide an overview of the approach developed by our team in collaboration with our community partners.

show abstract

Section: A Challenges In Data Processingmentioning

confidence: 99%

Designing Decision Support Systems for Emergency Response: Challenges and Opportunities

Pettet¹,

Baxter²,

Vazirizade³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…WildfireDB is a tool for modeling fire occurrence and spread by analyzing data about locations, sizes, vegetation, field, and other topographic features [16]. The architecture of the tool is shown in Figure 2.…”

Section: Workload Descriptionmentioning

confidence: 99%

URegM: a unified prediction model of resource consumption for refactoring software smells in open source cloud

Imran

Kosar

2022

2022 the 3rd European Symposium on Software Engineering

View full text Add to dashboard Cite

The low cost and rapid provisioning capabilities have made the cloud a desirable platform to launch complex scientific applications. However, resource utilization optimization is a significant challenge for cloud service providers, since the earlier focus is provided on optimizing resources for the applications that run on the cloud, with a low emphasis being provided on optimizing resource utilization of the cloud computing internal processes. Code refactoring has been associated with improving the maintenance and understanding of software code. However, analyzing the impact of the refactoring source code of the cloud and studying its impact on cloud resource usage require further analysis. In this paper, we propose a framework called Unified Regression Modelling (URegM) which predicts the impact of code smell refactoring on cloud resource usage. We test our experiments in a real-life cloud environment using a complex scientific application as a workload. Results show that URegM is capable of accurately predicting resource consumption due to code smell refactoring. This will permit cloud service providers with advanced knowledge about the impact of refactoring code smells on resource consumption, thus allowing them to plan their resource provisioning and code refactoring more effectively.CCS Concepts: • Software and its engineering → Maintaining software.

show abstract

“…Zonal statistics is an example of zonal operations. This method in RDPro is implemented using our previous work called RJ ⋊ ⋉ [69].…”

Section: Chaptermentioning

confidence: 99%

Raptor: Large Scale Processing of Big Raster + Vector Data

Singla

2021

Proceedings of the 2021 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

Advancements in remote sensing technology have resulted in petabytes of remote sensing data being made publicly available. The widespread use of smart devices and GPS technology has also led to the availability of highly accurate geographical features. This increase in the amount of spatial data has allowed for greater research opportunities in many scientific domains including hydrology, political science, environmental science, and agriculture. In these applications, scientists rarely base all their analysis on a single dataset but they usually need to combine multiple datasets in their analysis. Machine learning is a popular tool used by these scientists and often requires combining different datasets into a form usable by the machine learning algorithms. Spatial data is generally available in two representations, raster, and vector. The best data science and machine learning applications need to combine multiple datasets of both representations which is a data and compute-intensive problem.My dissertation proposes a new system called Raptor that bridges the gap between raster and vector data. It is an end-to-end system for efficiently processing raster and vi vector geospatial data concurrently. First, it discusses an initial approach to parallelize the zonal statistics operation called DARaptor. Second, it proposes Raptor Zonal Statistics, a system implemented in Hadoop that can be used to perform the zonal statistics operation for big raster and vector datasets. Third, it proposes Raptor Join which is modeled as a relational join operator in Spark that can be easily combined with other operators, while also offering the advantage of in-situ processing. Raptor Join is flexible to support adhoc applications and has been used for various real-world applications such as wildfire modeling, area interpolation, and crop yield mapping. Finally, this work proposes RDPro to add distributed raster pre-processing capabilities to Raptor that can scale to big data.The experimental evaluation on large-scale satellite data with up to a trillion pixels, and big vector data with up to hundreds of millions of segments and billions of points has shown that the proposed system is promising and can scale to big data. vii

show abstract

The Raptor Join Operator for Processing Big Raster + Vector Data

Cited by 11 publications

References 42 publications

Designing Decision Support Systems for Emergency Response: Challenges and Opportunities

Designing Decision Support Systems for Emergency Response: Challenges and Opportunities

URegM: a unified prediction model of resource consumption for refactoring software smells in open source cloud

Raptor: Large Scale Processing of Big Raster + Vector Data

Contact Info

Product

Resources

About