Efficiently Integrating Mobility and Environment Data for Climate Change Analytics

Jawarneh, Isam Mashhour Al; Bellavista, Paolo; Corradi, Antonio; Montanari, Rebecca

doi:10.1109/camad52502.2021.9617784

Cited by 7 publications

(7 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worth mentioning that those geocodes are the join keys. Readers are referred to [1] for detailed information exhibiting how this method works. Given the forementioned overarching traits of the filter-and-refine approach, we are adapting it in our system design in this paper.…”

Section: A Spark Structured Streaming and Distributed Geospatialmentioning

confidence: 99%

“…Spatial join is a prohibitively expensive operation which can render most advanced SPE systems irresponsive during spikes in data loads [1]. To alleviate this problem, various works in the relevant literature have designed frameworks for controlling the arrival rates of data coming from streaming sources.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, most effective dynamic application scenarios, such as those recurring in smart cities, require intermixing various workloads in a mashup fashion [1] by joining data-atrest with streaming data to pluck required insights. This operation, known as stream-static join, where one side of the join is a data stream, while the other is a static file residing in disk or in fast memory, is naturally expensive and may easily turn computationally prohibitive [1], particularly in distributed computing deployments where cross-network data shuffling is normally involved.…”

Section: Introductionmentioning

confidence: 99%

“…For computing the three constituent terms that are comprising the PID (Proportional, Integrative and Derivative terms), we have exploited an analogous mathematical model-based method similar to the one that has been employed in the plain Spark Streaming version. That said, after each batch interval, the new desirable rate is computed using (1). 𝑟𝑎𝑡𝑒 𝑛𝑒𝑤 = 𝑟𝑎𝑡𝑒 𝑙𝑎𝑡𝑒𝑠𝑡 − ((𝑃. 𝑒𝑟𝑟) + (𝐼. 𝑒𝑟𝑟 ℎ𝑖𝑠𝑡 ) + (𝐷. 𝑒𝑟𝑟 𝑑 )) (1) where (𝑃. 𝑒𝑟𝑟), (𝐼. 𝑒𝑟𝑟 ℎ𝑖𝑠𝑡 ) and (𝐷. 𝑒𝑟𝑟 𝑑 ) are the proportional, integrative, and derivative terms in the PID controller, respectively.…”

mentioning

confidence: 99%

“…That said, after each batch interval, the new desirable rate is computed using (1). 𝑟𝑎𝑡𝑒 𝑛𝑒𝑤 = 𝑟𝑎𝑡𝑒 𝑙𝑎𝑡𝑒𝑠𝑡 − ((𝑃. 𝑒𝑟𝑟) + (𝐼. 𝑒𝑟𝑟 ℎ𝑖𝑠𝑡 ) + (𝐷. 𝑒𝑟𝑟 𝑑 )) (1) where (𝑃. 𝑒𝑟𝑟), (𝐼. 𝑒𝑟𝑟 ℎ𝑖𝑠𝑡 ) and (𝐷. 𝑒𝑟𝑟 𝑑 ) are the proportional, integrative, and derivative terms in the PID controller, respectively. The approach through which every term is computed is adapted from the plain PID application as it appears in [11].…”

mentioning

confidence: 99%

See 4 more Smart Citations

SpatialSSJP: QoS-Aware Adaptive Approximate Stream-Static Spatial Join Processor

Jawarneh,

Bellavista,

Corradi

et al. 2024

IEEE Trans. Parallel Distrib. Syst.

Self Cite

View full text Add to dashboard Cite

The widespread adoption of Internet of Things (IoT) motivated the emergence of mixed workload scenarios in smart cities, where fast arriving geo-referenced massive amounts data streams need to be joined with archive tables, at scale. This aims at enriching streams with descriptive attributes that enable deeper insightful analytics. More applications are now relying on finding, in real-time, to which geographical region each data streaming spatially-tagged tuple belongs. This problem requires a computationally intensive stream-static join operation, where one side of join is a dynamic stream while the other is a diskresident static table. Even with emergence of some libraries that solve this problem in static-static fashion, their adoption for live scenarios is challenging because join operations are expensive in real-time. In addition, the time-varying nature of fluctuation and skewness in the geospatial data loads arriving online calls for an approximate solution that can trade-off QoS constraints in a way which ensures that the system survives sudden spikes in data loads. In this paper, we present SpatialSSJP, an adaptive spatialaware approximate query processing system that specifically focuses on stream-static joins in a way that guarantees achieving an agreed set of Quality-of-Service goals and maintains geostatistics of stateful online aggregations over stream-static join results. SpatialSSJP employs a state-of-art stratified-like sampling design to select well-balanced representative geospatial data stream samples and serve them to a stream-static geospatial join operator downstream. We implemented a prototype atop Spark Structured Streaming. Our extensive evaluations on big real datasets show that our system can survive and mitigate harsh join workloads and outperform state-of-art baselines by significant magnitudes, without risking rigorous error bounds in terms of the accuracy of the output results. SpatialSSJP achieves a relative accuracy gain against plain Spark joins of approximately 10% in worst cases but reaching up to 50% in best case scenarios.

show abstract

Section: A Spark Structured Streaming and Distributed Geospatialmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%