Automating Characterization Deployment in Distributed Data Stream Management Systems

Wang, Chunkai; Meng, Xiaofeng; Guo, Qi; Weng, Zujian; Yang, Chen

doi:10.1109/tkde.2017.2751606

Cited by 24 publications

(21 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Past works focused on different aspects of the problem: [38] investigates the scheduling problem satisfying load balancing and cost considerations; [68] employs design-time knowledge and benchmarking method to deal with scheduling on heterogeneous clusters; Re-Stream [83] focuses on energy-efficient resource scheduling; [93] addresses adaptive scheduling. Others focus on automatically optimizing and tuning workloads using various techniques such as cost-based (e.g., [43,50,87]) or ML-based (e.g., [16,89,90]).…”

Section: Cloud Resource Management and Tuningmentioning

confidence: 99%

S2ce

Kourtellis

Herodotou

Grzenda

et al. 2021

Proceedings of the 15th ACM International Conference on Distributed and Event-Based Systems

View full text Add to dashboard Cite

The explosive increase in volume, velocity, variety, and veracity of data generated by distributed and heterogeneous nodes such as IoT and other devices, continuously challenge the state of art in big data processing platforms and mining techniques. Consequently, it reveals an urgent need to address the ever-growing gap between this expected exascale data generation and the extraction of insights from these data. To address this need, this paper proposes Stream to Cloud & Edge (S2CE), a first of its kind, optimized, multi-cloud and edge orchestrator, easily configurable, scalable, and extensible. S2CE will enable machine and deep learning over voluminous and heterogeneous data streams running on hybrid cloud and edge settings, while offering the necessary functionalities for practical and scalable processing: data fusion and preprocessing, sampling and synthetic stream generation, cloud and edge smart resource management, and distributed processing. CCS CONCEPTS• Information systems → Data stream mining; • Computer systems organization → Cloud computing; • Computing methodologies → Machine learning.

show abstract

Section: Cloud Resource Management and Tuningmentioning

confidence: 99%

S2ce

Kourtellis

Herodotou

Grzenda

et al. 2021

Proceedings of the 15th ACM International Conference on Distributed and Event-Based Systems

View full text Add to dashboard Cite

show abstract

“…Users will submit their applications to the DSPS in the form of a query plan of certain format. The query plan is immediately converted into a directed acyclic graph (DAG) consisting of tasks which are running on the DSPS [8]. DSPS include three main data processing levels and they are:…”

Section: Distributed Stream Processing Systems (Dsps)mentioning

confidence: 99%

Component Profiling and Prediction Models for QoS-Aware Self-Adapting DSMS Framework

Yagnik

Chen

Kasraian

2021

2021 5th International Conference on Cloud and Big Data Computing (ICCBDC)

View full text Add to dashboard Cite

Quality of Service (QoS) has been identified as an important attribute of system performance of Data Stream Management Systems (DSMS). A DSMS should have the ability to allocate physical computing resources between different submitted queries and fulfil QoS specifications in a fair and square manner. System scheduling strategies need to be adjusted dynamically to utilise available physical resources to guarantee the end-to-end quality of service levels. In this paper, we present a proactive method that utilises a multi-level component profiling approach to build prediction models that anticipate several QoS violations and performance degradations. The models are constructed using several incremental machine learning algorithms that are enhanced with ensemble learning and abnormal detection techniques. The approach performs accurate predictions in near real-time with accuracy up to 85% and with abnormal detection techniques, the accuracy reaches 100%. This is a major component within a proposed QoS-Aware Self-Adapting Data Stream Management Framework.

show abstract

“…OrientStream [117] is a framework that uses incremental machine learning approaches for modeling and predicting the resource usage (i.e., CPU, memory, latency, and throughput) of workloads in distributed stream processing systems. Specifically, OrientStream uses an ensemble of four incremental learning models-namely, Naive Bayes, HoeffdingTree, Online bagging, and Nearest neighbors, which are trained using a variety of data-, plan-, operator-, and cluster-level features.…”

Section: Stream Processing Systemsmentioning

confidence: 99%

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Herodotou

Chen

2020

ACM Comput. Surv.

View full text Add to dashboard Cite

Big data processing systems (e.g., Hadoop, Spark, Storm) contain a vast number of configuration parameters controlling parallelism, I/O behavior, memory settings, and compression. Improper parameter settings can cause significant performance degradation and stability issues. However, regular users and even expert administrators grapple with understanding and tuning them to achieve good performance. We investigate existing approaches on parameter tuning for both batch and stream data processing systems and classify them into six categories: rule-based, cost modeling, simulation-based, experiment-driven, machine learning, and adaptive tuning. We summarize the pros and cons of each approach and raise some open research problems for automatic parameter tuning.

show abstract

Automating Characterization Deployment in Distributed Data Stream Management Systems

Cited by 24 publications

References 24 publications

S2ce

S2ce

Component Profiling and Prediction Models for QoS-Aware Self-Adapting DSMS Framework

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Contact Info

Product

Resources

About