Performance Analysis of Storm in a Real-World Big Data Stream Computing Environment

Huang, Yan; Sun, Dawei; Gao, Shang; Zhou, Zhangbing

doi:10.1007/978-3-030-00916-8_57

Cited by 4 publications

(11 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…26. From the average throughput of the five configurations (5,8,12), (5,10,10), (5,12,8) and (7,10,8), (7,8,10), we find that when the number of threads assigned to the launching component stays constant, the average throughput in one hour remains the same regardless of how the two Bolt components are changed. When the number of threads assigned to Spout component increases, the system throughput also increases; when the number of threads assigned to Spout component decreases, the system throughput also decreases.…”

Section: Resultsmentioning

confidence: 96%

“…When the number of launching component is unchanged, the average throughput is constant regardless of how the three Bolt components are changed. In terms of system latency, for the WordCount topology, when the number of threads assigned to the three components is (5,8,12), (5,12,8), (5,10,10), (9,7,9), (12,5,8) , the data processing delay is maintained at 3.5 milliseconds in one hour as shown in Fig. 28.…”

Section: Resultsmentioning

confidence: 99%

“…Secondly, for a given number of Workers in each topology, the number of threads increases in turn. In the WordCount Topology, when the first test sets the number of Workers to 3, the numbers of threads are (5,8,12), (10,16,24), (15,24,36), (20,34,48), (25,40,60) Two sets of experiments test twenty sets of data for each topology and we then analyze the effect of changes made to the number of threads assigned to the three/four components on the throughput of the Storm system and the processing delay for a given a certain number of Workers.…”

Section: Experimental Processmentioning

confidence: 99%

“…Finally, the number of processes set in the topologies remains the same, the total number of threads of all components keeps unchanged, while the number of threads of each component is adjusted. When the number of Workers is set to 3, and the number of threads in the three components is (5,8,12), (5,10,10), (5,12,8) In order to analyze the impact of hardware performance on Storm system operation, two scenario are designed as follows:…”

Section: Experimental Processmentioning

confidence: 99%

“…The performance of Storm plays a crucial role in many different application scenarios, however few studies were conducted to evaluate the corresponding performance of Storm [10] [11] [12]. We have been developing real-time big data processing applications on Storm for years, and deeply understand the importance of improving the processing efficiency.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Performance Evaluation and Analysis of Multiple Scenarios of Big Data Stream Computing on Storm Platform

2018

KSII TIIS

View full text Add to dashboard Cite

In big data era, fresh data grows rapidly every day. More than 30,000 gigabytes of data are created every second and the rate is accelerating. Many organizations rely heavily on real time streaming, while big data stream computing helps them spot opportunities and risks from real time big data. Storm, one of the most common online stream computing platforms, has been used for big data stream computing, with response time ranging from milliseconds to sub-seconds. The performance of Storm plays a crucial role in different application scenarios, however, few studies were conducted to evaluate the performance of Storm. In this paper, we investigate the performance of Storm under different application scenarios. Our experimental results show that throughput and latency of Storm are greatly affected by the number of instances of each vertex in task topology, and the number of available resources in data center. The fault-tolerant mechanism of Storm works well in most big data stream computing environments. As a result, it is suggested that a dynamic topology, an elastic scheduling framework, and a memory based fault-tolerant mechanism are necessary for providing high throughput and low latency services on Storm platform.

show abstract

Section: Resultsmentioning

confidence: 96%

Section: Resultsmentioning

confidence: 99%

Section: Experimental Processmentioning

confidence: 99%

Section: Experimental Processmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Performance Evaluation and Analysis of Multiple Scenarios of Big Data Stream Computing on Storm Platform

2018

KSII TIIS

View full text Add to dashboard Cite

show abstract

Maximum Sustainable Throughput Evaluation Using an Adaptive Method for Stream Processing Platforms

Chu

Hamdull

2020

IEEE Access

View full text Add to dashboard Cite

The volume and type of streaming data is increasing rapidly, thus, real-time processing scenarios for streaming data have continued to increase. The inherent volatility of streaming data causes large changes in the throughput volatility of stream processing platforms, making evaluations of the maximum sustainable throughput (MST) for such platforms a challenge. To address these problems and improve the low efficiency of manual evaluations, a naïve method is typically used to evaluate the MST of platforms by periodically measuring throughput. The throughput volatility is detected using a skip sliding window that is defined as a bounded sequence in which a limited number of throughput values stored to measure volatility in a skipping fashion in each data growth cycle, which is a fixed time interval with respect to the continuous data rate increases. A skipping sliding window that is used to describe the procedure with which the MST is calculated in an approximate way is based on a sliding window, which is used usually in the context of stateful streaming operators. Then, according to the throughput volatility, the method judges whether the MST has been reached using a latency guarantee. However, because this method has low efficiency and a large error rate, we propose an adaptive MST evaluation method that adds a data-growth factor function to the naïve method cycle that dynamically and adaptively tunes the data rate for each data growth cycle. The experimental results from four open-source benchmarks running on three mainstream stream processing platforms show that the adaptive MST evaluation method has a lower error rate and executes faster compared with the naïve evaluation method. Moreover, the proposed method is insensitive to parameter variations and is suitable for mainstream stream processing platforms.

show abstract

Performance Prediction Method for Stream Computing Platform Based on Time Series

Lü

2021

IEEE Access

View full text Add to dashboard Cite

As one of the most popular high-performance data processing technology, existing task and resource scheduling strategies for stream computing platforms are suffering from the problem of triggering hysteresis, which seriously affects the cluster performance. To address this problem, the idea of performance prediction based on timeline series data is proposed. Firstly, the performance variation rule of the stream processing platform is analyzed, which provides a basis for proposing the performance prediction model. Secondly, the basic topology paradigm, periodic load prediction model, and real-time throughput prediction model are proposed as the theoretical foundation for performance prediction. Thirdly, the performance prediction algorithm is proposed to predict the variation trend of the processing load and throughput of the cluster. The processing load is predicted periodically while the throughput of the cluster is predicted in a real-time manner. Finally, the performance evaluation algorithm is proposed to evaluate the cluster performance based on the prediction results and trigger the corresponding scheduling strategies in advance. The experimental results showed that the prediction accuracy of the proposed algorithm meets the requirements in practical applications. Meanwhile, the proposed method improves the performance of stream computing performance by triggering the scheduling strategies in advance.INDEX TERMS Big data, stream computing, performance prediction, processing load, and throughput.

show abstract

Performance Analysis of Storm in a Real-World Big Data Stream Computing Environment

Cited by 4 publications

References 12 publications

Performance Evaluation and Analysis of Multiple Scenarios of Big Data Stream Computing on Storm Platform

Performance Evaluation and Analysis of Multiple Scenarios of Big Data Stream Computing on Storm Platform

Maximum Sustainable Throughput Evaluation Using an Adaptive Method for Stream Processing Platforms

Performance Prediction Method for Stream Computing Platform Based on Time Series

Contact Info

Product

Resources

About