Evaluating Micro-batch and Data Frequency for Stream Processing Applications on Multi-cores

Garcia, Adriano Marques; Griebler, Dalvan; Schepke, Claudio; Fernandes, Luiz Gustavo

doi:10.1109/pdp55904.2022.00011

Cited by 5 publications

(6 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In FastFlow, each thread executes the same operator/stage on several items throughout the execution. This model adds static communication queues between the stages and, if there is no load balancing among the stages (which is the case with Ferret), this can impact the application performance [14]. It explains the higher latency of FastFlow compared to TBB.…”

Section: Micro-batching Resultsmentioning

confidence: 99%

“…This paper extends this last previous work from [14]. We use the micro-batching system to evaluate the benchmarks on another architecture and use different parallel patterns.…”

Section: Introductionmentioning

confidence: 82%

“…In those works, we evaluated the performance of different real-world stream applications under multiple PPIs in terms of latency, throughput, and resource utilization. In more recent work [14], we evaluated micro-batching and data frequency, but we only evaluated one parallel pattern, the frequency patterns were not representative, and the analysis was quite limited by space issues. Also, the metrics used there were not enough to bring out some important aspects of the benchmarks.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Micro-batch and data frequency for stream processing on multi-cores

et al. 2023

Self Cite

View full text Add to dashboard Cite

Latency or throughput are often critical performance metrics in stream processing. Applications' performance can fluctuate depending on the input stream. This unpredictability is tied to variations in data arrival frequency, data size, complexity, and other factors. Researchers are constantly investigating new ways to mitigate the impact of these variations on performance with self-adaptive techniques involving elasticity or micro-batching. However, there is a lack of benchmarks capable of creating test scenarios to further evaluate these techniques. This work extends and improves the SPBench benchmarking framework to support dynamic micro-batching and data stream frequency management. We also propose a set of algorithms that generate the most commonly used frequency patterns for benchmarking stream processing in related work. It allows the creation of a wide variety of test scenarios. To validate our solution, we use SPBench to create custom benchmarks and evaluate the impact of micro-batching and data stream Micro-batch and Data Frequency for Stream Processing on Multi-cores frequency on the performance of Intel TBB and FastFlow. These are two libraries that leverage stream parallelism for multi-core architectures. Our results demonstrated that most test cases benefited from micro-batches, especially high throughput applications with ordering constraints. For different data stream frequency configurations, TBB ensured the lowest latency, while FastFlow assured higher throughput in shorter pipelines.

show abstract

Section: Micro-batching Resultsmentioning

confidence: 99%

“…This paper extends this last previous work from [14]. We use the micro-batching system to evaluate the benchmarks on another architecture and use different parallel patterns.…”

Section: Introductionmentioning

confidence: 82%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Micro-batch and data frequency for stream processing on multi-cores

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although published years before all modern stream processing frameworks considered in this work have been released, it is still used in research [77,82,83] and compared to newer benchmarks [9,63]. Pagliari et al [84] and Garcia et al [85,86] present approaches to generate benchmarks.…”

Section: Related Work On Stream Processing Benchmarksmentioning

confidence: 99%

Benchmarking Scalability of Stream Processing Frameworks Deployed as Event-Driven Microservices in the Cloud

Henning,

Hasselbring

2023

Preprint

View full text Add to dashboard Cite

“…Um exemplo de um Benchmark deste tipo pode ser encontrado em [Bordin et al 2020]. A implementação de técnicas específicas de programação para obter melhor desempenho de uma aplicação de stream de dados, também são encontradas na literatura como a ideia em [Garcia et al 2022]. Vale notar que o [Garcia et al 2022] também foca-se em um sistema multicore e não em um sistema distribuído.…”

Section: Trabalhos Relacionadosunclassified

Um estudo sobre uso do MPI para uma aplicação de detecção de picos em data streams

Müller

Griebler

2023

Anais Da XXIII Escola Regional De Alto Desempenho Da Região Sul (ERAD-RS 2023)

View full text Add to dashboard Cite

Aplicações de data stream podem ser implementadas com diferentes interfaces de programação paralela. Neste artigo, realizou-se um estudo e implementação da aplicação Spike Detection com MPI e a comparou-se com versões usando Flink, Storm e Windflow. Avaliou-se o trouhgput e conclui-se que a implementação com Windflow apresenta o melhor desempenho, enquanto as versões com MPI tiveram um throughput inferior as demais soluções.

show abstract

Evaluating Micro-batch and Data Frequency for Stream Processing Applications on Multi-cores

Cited by 5 publications

References 23 publications

Micro-batch and data frequency for stream processing on multi-cores

Micro-batch and data frequency for stream processing on multi-cores

Benchmarking Scalability of Stream Processing Frameworks Deployed as Event-Driven Microservices in the Cloud

Um estudo sobre uso do MPI para uma aplicação de detecção de picos em data streams

Contact Info

Product

Resources

About