Rethinking Stateful Stream Processing with RDMA

Monte, Bonaventura Del; Zeuch, Steffen; Rabl, Tilmann; Markl, Volker

doi:10.1145/3514221.3517826

Cited by 8 publications

(4 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This has the potential to mitigate the bottleneck that networks pose in distributed settings while fulfilling the realtime requirements of stream processing. With a novel processing model designed for RDMA, Slash [21] accelerates distributed stream processing computations. The stateful query executor applies multiple instances of the same operator in parallel to scale the processing of streaming queries across many nodes.…”

Section: Stream Slicingmentioning

confidence: 99%

“…In the evaluation presented by Del Monte et al [21], Slash is compared against state-of-the-art stream processing systems such as LightSaber [58] and Apache Flink [13]. Native RDMA acceleration allows the system to scale with the number of nodes and enables higher throughput for common stream workloads than partitioning-based approaches.…”

Section: Algorithms For Optimizations On Modern Hardwarementioning

confidence: 99%

“…The approaches presented in Sect. 4 exploit advantages of modern hardware, such as multi-core processors and high- Cutty [14], Scotty [65,66,68], AJoin [34], Eco-Join [44], Zhang et al [76], Parallel ADWIN [26] Modern Hardware Zeuch et al [72], Grizzly [28], Slash [21] Distributed Window Processing AStream [35], Disco [8], Rhino [6,20] Stream Processing Systems Scotty Connectors to Apache Flink [13], Storm [61], Beam [3], Samza [47], Kafka Streams [38], Spark [60,71]; NebulaStream [73,74], Agora [69], SENSE [62,63,67] Applications…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Algorithms for Windowed Aggregations and Joins on Distributed Stream Processing Systems

et al. 2022

Self Cite

View full text Add to dashboard Cite

Window aggregations and windowed joins are central operators of modern real-time analytic workloads and significantly impact the performance of stream processing systems.This paper gives an overview of state-of-the-art research in this area conducted by the Berlin Institute for the Foundations of Learning and Data (BIFOLD) and the Technische Universität Berlin. To this end, we present different algorithms for efficiently processing windowed operators and discuss techniques for distributed stream processing. Recently, several approaches have leveraged modern hardware for windowed stream processing, which we will also include in this overview. Additionally, we describe the integration of windowed operators into various stream processing systems and diverse applications that use specialized window operations.

show abstract

Section: Stream Slicingmentioning

confidence: 99%

Section: Algorithms For Optimizations On Modern Hardwarementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Algorithms for Windowed Aggregations and Joins on Distributed Stream Processing Systems

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…AMNES operates specifically on the Converged Ethernet (RoCEv2) protocol, which is already deployed by many cloud providers such as Alibaba Cloud [24], Microsoft Azure [26], with 70% of Azure traffic being RDMA-based [54], and Oracle Exadata [59]. RDMA has also made its way into database design, with increasing systems and prototypes demonstrating its advantages [55,82].…”

Section: Introductionmentioning

confidence: 99%

AMNES: Accelerating the Computation of Data Correlation Using FPGAs

Chiosa,

Preußer,

Blott

et al. 2023

Proc. VLDB Endow.

View full text Add to dashboard Cite

A widely used approach to characterize input data in both databases and ML is computing the correlation between attributes. The operation is supported by all major database engines and ML platforms. However, it is an expensive operation as the number of attributes involved grows. To address the issue, in this paper we introduce AMNES, a stream analytics system offloading the correlation operator into an FPGA-based network interface card. AMNES processes data at network line rate and the design can be used in combination with smart storage or SmartNICs to implement near data or in-network data processing. AMNES design goes beyond matrix multiplication and offers a customized solution for correlation computation bypassing the CPU. Our experiments show that AMNES can sustain streams arriving at 100 Gbps over an RDMA network, while requiring only ten milliseconds to compute the correlation coefficients among 64 streams, an order of magnitude better than competing CPU or GPU designs.

show abstract

Survey of window types for aggregation in stream processing systems

et al. 2023

View full text Add to dashboard Cite

In this paper, we present the first comprehensive survey of window types for stream processing systems which have been presented in research and commercial systems. We cover publications from the most relevant conferences, journals, and system whitepapers on stream processing, windowing, and window aggregation which have been published over the last 20 years. For each window type, we provide detailed specifications, formal notations, synonyms, and use-case examples. We classify each window type according to categories that have been proposed in literature and describe the out-of-order processing. In addition, we examine academic, commercial, and open-source systems with respect to the window types that they support. Our survey offers a comprehensive overview that may serve as a guideline for the development of stream processing systems, window aggregation techniques, and frameworks that support a variety of window types.

show abstract

Rethinking Stateful Stream Processing with RDMA

Cited by 8 publications

References 49 publications

Algorithms for Windowed Aggregations and Joins on Distributed Stream Processing Systems

Algorithms for Windowed Aggregations and Joins on Distributed Stream Processing Systems

AMNES: Accelerating the Computation of Data Correlation Using FPGAs

Survey of window types for aggregation in stream processing systems

Contact Info

Product

Resources

About