Influence of Learning Rate Hyper-Parameter and Early Stopping Training Technique in Training Vanilla Neural Network Model for Effective Signal Power Loss Prediction

Abstract

In this research work, the authors employed a Vanilla Neural Network (VNN) model to examine the influence of the Learning Rate (LR) hyper-parameter, early stopping training technique, Levenberg-Marquardt (LM) and Bayesian Regularization (BR) training algorithms in the training and prediction of signal power loss using a measured dataset from a Long Term Evolution (LTE) micro-cell environment. First order statistical performance indices, including the Mean Squared Error (MSE), Root Mean Square Error (RMSE), and Regression (R), were adopted for interpreting and analyzing the results. The LR hyper-parameters were sequentially selected cyclically from 0.002 to 1.00, while the early stopping training technique was chosen at a ratio of 70%:15%:15% for training, testing, and validation of the neural network model during network training. The authors examined the neural network training and its predictive abilities of the measured signal power by training the model with LM and BR training algorithms and applying varied LR values and the early stopping training technique. Comparative analysis was also conducted by training the VNN model without the application of LR hyper-parameter and the early stopping training technique. The LR hyper-parameter is a distinct training parameter that, when efficiently applied, improves network convergence, while the application of training techniques such as early stopping minimizes over-fitting during network training. The training result outputs demonstrate the effectiveness of the VNN model when applying a very small LR of 0.002. The best prediction results of the VNN model were observed when using an LR of 0.002, with an R value of 0.9922, performance MSE of 1.48, and RMSE of 1.6790 while training the VNN model with the BR algorithm and applying the early stopping training technique. When training the VNN model with an LR of 0.002 using the LM algorithm, an R value of 0.9816, performance MSE of 5.060, and RMSE of 2.5619 were obtained. The worst prediction results were computed when training the VNN model without the application of LR and the early stopping training technique, resulting in an R value of 0.97480, performance MSE of 6.3, and RMSE of 3.0167. However, when training the same VNN model without the application of LR and the early stopping training technique using the BR training algorithm, an R value of 0.9910, performance MSE of 3.3, and RMSE of 1.8045 were obtained. This strongly demonstrates that the BR training algorithm is not only a good training algorithm but also an excellent training technique.