Apache Flink: Stream Analytics at Scale

Katsifodimos, Asterios; Schelter, Sebastian

doi:10.1109/ic2ew.2016.56

Cited by 54 publications

(30 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DSP systems. Various DSP systems such as Apache Storm [2], Flink [3], and Spark Streaming [4] have been proposed to develop and execute stream data applications. They are usually built on a cluster or cloud, combining resources of multiple physical servers to process data streams continuously.…”

Section: A Stream Data Processingmentioning

confidence: 99%

“…It is predicted that there will be 50 billions interconnected Internet of Thing (IoT) devices which are expected to generate 400 Zetta Bytes of data per year by 2020 [1]. This led to the proliferation of Distributed Stream Processing (DSP) systems such as Storm [2], Flink [3], and Spark Streaming [4] in data centers and clouds to perform online processing of these continuous and unbounded data streams [5]. Despite that clouds can provide abundant computing resources to meet the ever-growing computation demands of streaming data applications, processing stream data in the clouds may come with high latency -this strongly depends on the transmission time from the data source to the clouds.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

When FPGA-Accelerator Meets Stream Data Processing in the Edge

Ibrahim

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

Today, stream data applications represent the killer applications for Edge computing: placing computation close to the data source facilitates real-time analysis. Previous efforts have focused on introducing lightweight distributed stream processing (DSP) systems and dividing the computation between Edge servers and the clouds. Unfortunately, given the limited computation power of Edge servers, current efforts may fail in practice to achieve the desired latency of stream data applications. In this vision paper, we argue that by introducing FPGAs in Edge servers and integrating them into DSP systems, we might be able to realize stream data processing in Edge infrastructures. We demonstrate that through the design, implementation, and evaluation of F-Storm, an FPGA-accelerated and general-purpose distributed stream processing system on Edge servers. F-Storm integrates PCIe-based FPGAs into Edge-based stream processing systems and provides accelerators as a service for stream data applications. We evaluate F-Storm using different representative stream data applications. Our experiments show that compared to Storm, F-Storm reduces the latency by 36% and 75% for matrix multiplication and grep application. It also obtains 1.4x and 2.1x improvement for these two applications, respectively. We expect this work to accelerate progress in this domain.

show abstract

Section: A Stream Data Processingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

When FPGA-Accelerator Meets Stream Data Processing in the Edge

Ibrahim

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

show abstract

“…[20]. This broad adoption results in the proliferative demand of stream data processing frameworks such as Spark * The corresponding author Streaming [28], Flink [12], Storm [22]. Different stream data processing frameworks provide different advantages, e.g., high throughput or low latency.…”

Section: Introductionmentioning

confidence: 99%

“…In particular, there are two main types of paradigms in stream processing, i.e., data stream and operation stream. Data stream is widely adopted by most existing stream processing frameworks, e.g., S4 [17], MillWheel [1], Naiad [16], Samza [19], Storm [22], Flink [12], and Heron [14]. Data stream fixes operations in certain workers and schedules data to flow through these operations, as illustrated in Figure 1(a).…”

Section: Introductionmentioning

confidence: 99%

“…That is to say, the processing paradigm of each operation should be skillfully selected according to its number of records. Notice that some frameworks like Flink [12] and Structured Streaming [9] in Spark now are also trying to support both data stream and operation stream, but they only implement a single paradigm in one stream application and cannot handle the data burst between the operations.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dual-Paradigm Stream Processing

Liu

Ibrahim

et al. 2018

Proceedings of the 47th International Conference on Parallel Processing

View full text Add to dashboard Cite

Existing stream processing frameworks operate either under data stream paradigm processing data record by record to favor low latency, or under operation stream paradigm processing data in micro-batches to desire high throughput. For complex and mutable data processing requirements, this dilemma brings the selection and deployment of stream processing frameworks into an embarrassing situation. Moreover, current data stream or operation stream paradigms cannot handle data burst efficiently, which probably results in noticeable performance degradation. This paper introduces a dual-paradigm stream processing, called DO (Data and Operation) that can adapt to stream data volatility. It enables data to be processed in micro-batches (i.e., operation stream) when data burst occurs to achieve high throughput, while data is processed record by record (i.e., data stream) in the remaining time to sustain low latency. DO embraces a method to detect data bursts, identify the main operations affected by the data burst and switch paradigms accordingly. Our insight behind DO's design is that the trade-off between latency and throughput of stream processing frameworks can be dynamically achieved according to data communication among operations in a fine-grained manner (i.e., operation level) instead of framework level. We implement a prototype stream processing framework that adopts DO. Our experimental results show that our framework with DO can achieve 5x speedup over operation stream under low data stream sizes, and outperforms data stream on throughput by 2.1x to 3.2x under data burst.

show abstract