Parameter Optimization of Spark in Heterogeneous Environment Based on Hyperband

Chang, Dajun; Qiao, Zhangquan; Li, Li; Zheng, Qi

doi:10.1109/bdeim55082.2021.00048

Cited by 3 publications

(1 citation statement)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different parameter values have different impacts on the execution performance of Spark jobs. According to two principles: the main parameters affect the running performance of the task job and the resource scheduling in the process of the task job [38][39][40]. Therefore, it is very important to calculate and select some necessary parameters of Spark through cluster resources.…”

Section: Spark Parameter Selectionmentioning

confidence: 99%

Simulation Research on Fast Matching of Big Data Based on Spark

Song

Leng

et al. 2023

IEEE Access

View full text Add to dashboard Cite

To solve the problem of low efficiency in real-time processing and matching of CNAME records in massive DNS log data, a parallel AC automaton enhancement method based on Spark was proposed. The method is based on the Spark distributed cluster computing engine of Hadoop, which ensures the stability of massive DNS log data storage with high fault tolerance and 24-hour real-time processing. At the same time, the Spark distributed cluster uses the multi-thread parallel computing method combined with the improved AC automaton algorithm, which not only reduces the memory occupied by trie construction, but also improves the efficiency of rapid matching of CNAME records of massive DNS logs. Simulation results show that the proposed method can quickly match CNAME records of massive DNS log data. Compared with the original AC algorithm, the efficiency is significantly improved, and the time complexity and storage space are reduced.

show abstract

Section: Spark Parameter Selectionmentioning

confidence: 99%

Simulation Research on Fast Matching of Big Data Based on Spark

Song

Leng

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

QHB⁺: Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications

Jang,

Yoon,

Jung

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Apache Spark stands out as a well-known solution for big data processing because of its efficiency and rapid processing capabilities. One of its modules, Spark SQL, serves as a prominent big data query engine. However, executing Spark SQL applications with massive data can be time-intensive, and the execution time can vary significantly depending on its configurations. Recent studies try to reduce application execution times by searching optimal configurations for applications. While Bayesian optimization is recognized as a powerful method in recent studies for configuration optimization, it faces challenges such as computational costs and time-consuming computations, especially when dealing with large search spaces Due to these challenges, we propose QHB + , designed to rapidly search optimal configurations. QHB + utilizes the Successive Halving Algorithm-based optimization methods, performing well in hyperparameter optimization of machine learning models, for configuration optimization of Spark SQL applications. Through empirical evaluations against established benchmarks, we show the efficiency of QHB + , highlighting them as swift alternatives to conventional optimization method for optimizing Spark SQL configurations.

show abstract

Spark Configurations to Optimize Decision Tree Classification on UNSW-NB15

Bagui

Walauskis

DeRush

et al. 2022

BDCC

View full text Add to dashboard Cite

This paper looks at the impact of changing Spark’s configuration parameters on machine learning algorithms using a large dataset—the UNSW-NB15 dataset. The environmental conditions that will optimize the classification process are studied. To build smart intrusion detection systems, a deep understanding of the environmental parameters is necessary. Specifically, the focus is on the following environmental parameters: the executor memory, number of executors, number of cores per executor, execution time, as well as the impact on statistical measures. Hence, the objective was to optimize resource usage and minimize processing time for Decision Tree classification, using Spark. This shows whether additional resources will increase performance, lower processing time, and optimize computing resources. The UNSW-NB15 dataset, being a large dataset, provides enough data and complexity to see the changes in computing resource configurations in Spark. Principal Component Analysis was used for preprocessing the dataset. Results indicated that a lack of executors and cores result in wasted resources and long processing time. Excessive resource allocation did not improve processing time. Environmental tuning has a noticeable impact.

show abstract

Parameter Optimization of Spark in Heterogeneous Environment Based on Hyperband

Cited by 3 publications

References 2 publications

Simulation Research on Fast Matching of Big Data Based on Spark

Simulation Research on Fast Matching of Big Data Based on Spark

QHB⁺: Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications

Spark Configurations to Optimize Decision Tree Classification on UNSW-NB15

Contact Info

Product

Resources

About

Parameter Optimization of Spark in Heterogeneous Environment Based on Hyperband

Cited by 3 publications

References 2 publications

Simulation Research on Fast Matching of Big Data Based on Spark

Simulation Research on Fast Matching of Big Data Based on Spark

QHB+: Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications

Spark Configurations to Optimize Decision Tree Classification on UNSW-NB15

Contact Info

Product

Resources

About

QHB⁺: Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications