Hyperparameter Tuning of Machine Learning Algorithms Using Response Surface Methodology: A Case Study of ANN, SVM, and DBN

Abstract

This study applies response surface methodology (RSM) to the hyperparameter fine-tuning of three machine learning (ML) algorithms: artificial neural network (ANN), support vector machine (SVM), and deep belief network (DBN). The purpose is to demonstrate RSM effectiveness in maintaining ML algorithm performance while reducing the number of runs required to reach effective hyperparameter settings in comparison with the commonly used grid search (GS). The ML algorithms are applied to a case study dataset from a food producer in Thailand. The objective is to predict a raw material quality measured on a numerical scale. K-fold cross-validation is performed to ensure that the ML algorithm performance is robust to the data partitioning process in the training, validation, and testing sets. The mean absolute error (MAE) of the validation set is used as the prediction accuracy measurement. The reliability of the hyperparameter values from GS and RSM is evaluated using confirmation runs. Statistical analysis shows that (1) the prediction accuracy of the three ML algorithms tuned by GS and RSM is similar, (2) hyperparameter settings from GS are 80% reliable for ANN and DBN, and settings from RSM are 90% and 100% reliable for ANN and DBN, respectively, and (3) savings in the number of runs required by RSM over GS are 97.79%, 97.81%, and 80.69% for ANN, SVM, and DBN, respectively.

Application of Response Surface Methodology in Dance Health and Training Safety

“…Response surface methodology (RSM) is a test design method proposed by Box et al It is an optimization method that integrates test design and mathematical modeling [22][23][24][25][26][27]. By performing tests on representative local points, the functional relationship between factors and results in the global scope is regressed and fitted, and the optimal level value of each element is obtained.…”

Artificial Neural Network Hyperparameters Optimization: A Survey

“…17. Mansi Gupta, et al [79] [83] Applied (RSM) to fine-tune the hyperparameters of three machine learning algorithms: (SVM), (ANN), and (DBN). The goal was to show that RSM is more efficient than grid search in keeping ML algorithm performance while decreasing the number of the runs that needed in order to reach appropriate hyper-parameter values.…”

Machine Learning Application in G.I.S. and Remote Sensing: An Overview

“…Xu, Zhao, Wang, & Shi, 2022;Yang & Shami, 2020) It's an additional step to improve the accuracy and performance of the model(Pannakkong, Thiwa-Anont, Singthong, Parthanadee, & Buddhakulsomsiri, 2022). For example, selection of the best polynomial features in linear regression models, number of trees in a random forest, number of layers and neurons in a neural network, maximum depth in decision trees, and learning rate for gradient descent (Pannakkong et al, 2022). Some common hyper parameter tuning techniques are grid search, randomized search, Bayesian optimization, sequential model-based optimization, and genetic algorithms (Yang & Shami, 2020).…”