A survey on the evolution of stream processing systems

Fragkoulis, Marios; Carbone, Paris; Kalavri, Vasiliki; Katsifodimos, Asterios

doi:10.1007/s00778-023-00819-8

Cited by 20 publications

(3 citation statements)

References 117 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Stream processing frameworks perform operations such as filterings, transformations, or aggregations in near-real time on continuous streams of data [19]. State-of-the-art frameworks are designed for high throughput and low-latency processing, while also scaling with massive amounts of data [9,16]. To address these requirements, they run in a distributed fashion on commodity hardware.…”

Section: Distributed Stream Processingmentioning

confidence: 99%

“…State-of-the-art distributed stream processing frameworks such as Spark [4,39], Flink [6], Kafka Streams [31,38], or Hazelcast with its Jet engine [10] have gained widespread adoption over the last years not only for building data analytics pipelines, but also for implementing core business logic in software-based organizations [9,22]. Such frameworks support software engineers in building highly scalable, reliable, and efficient applications that process continuous data streams of massive volume.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Henning,

Vogel,

Leichtfried

et al. 2024

Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

Distributed stream processing frameworks help building scalable and reliable applications that perform transformations and aggregations on continuous data streams. This paper introduces Shuf-fleBench, a novel benchmark to evaluate the performance of modern stream processing frameworks. In contrast to other benchmarks, it focuses on use cases where stream processing frameworks are mainly employed for shuffling (i.e., re-distributing) data records to perform state-local aggregations, while the actual aggregation logic is considered as black-box software components. ShuffleBench is inspired by requirements for near real-time analytics of a large cloud observability platform and takes up benchmarking metrics and methods for latency, throughput, and scalability established in the performance engineering research community. Although inspired by a real-world observability use case, it is highly configurable to allow domain-independent evaluations. ShuffleBench comes as a ready-to-use open-source software utilizing existing Kubernetes tooling and providing implementations for four stateof-the-art frameworks. Therefore, we expect ShuffleBench to be a valuable contribution to both industrial practitioners building stream processing applications and researchers working on new stream processing approaches. We complement this paper with an experimental performance evaluation that employs ShuffleBench with various configurations on Flink, Hazelcast, Kafka Streams, and Spark in a cloud-native environment. Our results show that Flink achieves the highest throughput while Hazelcast processes data streams with the lowest latency.

show abstract

Section: Distributed Stream Processingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Henning,

Vogel,

Leichtfried

et al. 2024

Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

show abstract

“…Through a detailed examination of architectural design principles, data integration and management strategies, and performance evaluation methodologies, we provide a comprehensive overview of how TPS can be optimized to support the dynamic needs of the retail sector. In presenting case studies of successful TPS integration, we highlight the transformative potential of data visualization tools in unlocking actionable insights from complex transaction data, thereby empowering retailers to make informed strategic decisions [1]. Through this analysis, we aim to offer a roadmap for retail organizations seeking to harness the power of TPS in their digital transformation journey, highlighting both the challenges and opportunities that lie ahead.…”

Section: Introductionmentioning

confidence: 99%

Exploring Improvement of Business Performance of Transaction Processing System in Retail Sector

Xie

2024

AEMPS

View full text Add to dashboard Cite

This paper delves into the strategic integration of Transaction Processing Systems (TPS) within the digital transformation initiatives of the retail sector, focusing on enhancing business performance through advanced business modelling, information system modelling, and leveraging data visualization for business intelligence. We examine strategic frameworks such as the Value Discipline Model and the Agile methodology to align TPS integration with business goals, emphasizing the importance of agility and continuous improvement. The impact of TPS on retail operations is quantitatively assessed through key performance indicators, revealing significant enhancements in transaction processing times, inventory management, and financial reporting accuracy. Challenges including legacy system integration, data security, and skill gaps are addressed through practical solutions like APIs, encryption methods, and comprehensive training programs. Furthermore, we explore the architectural design principles of TPS, emphasizing scalability, resilience, and adaptability, alongside effective data integration and management strategies that ensure data consistency and accessibility. Performance evaluation methodologies are also discussed, employing mathematical models for predicting system behavior and optimizing throughput, latency, and reliability. Case studies highlight the transformative power of data visualization tools in retail, showcasing their impact on operational efficiency and decision-making. This comprehensive analysis underscores the pivotal role of TPS in driving digital transformation in the retail industry, offering a roadmap for organizations aiming to enhance their competitive edge in the digital era.

show abstract

Enhancing self-adaptation for efficient decision-making at run-time in streaming applications on multicores

Vogel,

Danelutto,

Torquati

et al. 2024

J Supercomput

View full text Add to dashboard Cite

Parallel computing is very important to accelerate the performance of computing applications. Moreover, parallel applications are expected to continue executing in more dynamic environments and react to changing conditions. In this context, applying self-adaptation is a potential solution to achieve a higher level of autonomic abstractions and runtime responsiveness. In our research, we aim to explore and assess the possible abstractions attainable through the transparent management of parallel executions by self-adaptation. Our primary objectives are to expand the adaptation space to better reflect real-world applications and assess the potential for self-adaptation to enhance efficiency. We provide the following scientific contributions: (I) A conceptual framework to improve the designing of self-adaptation; (II) A new decision-making strategy for applications with multiple parallel stages; (III) A comprehensive evaluation of the proposed decision-making strategy compared to the state-of-the-art. The results demonstrate that the proposed conceptual framework can help design and implement self-adaptive strategies that are more modular and reusable. The proposed decision-making strategy provides significant gains in accuracy compared to the state-of-the-art, increasing the parallel applications’ performance and efficiency.

show abstract

A survey on the evolution of stream processing systems

Cited by 20 publications

References 117 publications

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Exploring Improvement of Business Performance of Transaction Processing System in Retail Sector

Enhancing self-adaptation for efficient decision-making at run-time in streaming applications on multicores

Contact Info

Product

Resources

About