The volume and type of streaming data is increasing rapidly, thus, real-time processing scenarios for streaming data have continued to increase. The inherent volatility of streaming data causes large changes in the throughput volatility of stream processing platforms, making evaluations of the maximum sustainable throughput (MST) for such platforms a challenge. To address these problems and improve the low efficiency of manual evaluations, a naïve method is typically used to evaluate the MST of platforms by periodically measuring throughput. The throughput volatility is detected using a skip sliding window that is defined as a bounded sequence in which a limited number of throughput values stored to measure volatility in a skipping fashion in each data growth cycle, which is a fixed time interval with respect to the continuous data rate increases. A skipping sliding window that is used to describe the procedure with which the MST is calculated in an approximate way is based on a sliding window, which is used usually in the context of stateful streaming operators. Then, according to the throughput volatility, the method judges whether the MST has been reached using a latency guarantee. However, because this method has low efficiency and a large error rate, we propose an adaptive MST evaluation method that adds a data-growth factor function to the naïve method cycle that dynamically and adaptively tunes the data rate for each data growth cycle. The experimental results from four open-source benchmarks running on three mainstream stream processing platforms show that the adaptive MST evaluation method has a lower error rate and executes faster compared with the naïve evaluation method. Moreover, the proposed method is insensitive to parameter variations and is suitable for mainstream stream processing platforms.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
customersupport@researchsolutions.com
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.