ps-CALR: Periodic-Shift Cosine Annealing Learning Rate for Deep Neural Networks

Abstract

There are continued efforts to build on the performance of deep learning (DL) models in various fields of application. Developing new DL models continues to open unprecedented opportunities in diverse application areas despite the enormous resources required. Generally, the learning mechanism of DL models depends on the term "cost function" (CF) or "loss function" (LF), and DL models require varied hyperparameter settings and, precisely, parameters that can help the model to continually minimize the cost function until faster convergence, with better generalization over the data in the loss landscape, is assumed. The learning rate (LR) update seeks to find the optimal solution for DL models through relative cost function minimization. Therefore, selecting the appropriate LR is essential to the performance of DL models. The existing cosine annealing LR, despite its demonstration for fast model convergence, lacks complete loss landscape exploration of the flat minima, hence limiting its ability to model better generalization. To address this, the paper proposes a period-shift Cosine Annealing Learning Rate with warm-up epochs (ps-CALR) to perturb the LR update. Six different publicly available datasets were used to benchmark the proposed LR method by experimenting with custom DL (multilayer perceptron and convolutional neural networks) and pre-trained DL models. The proposed ps-CARL enhances model generalization and convergence, pushing the solution to notably better performance compared with fixed LR and the existing cosine annealing method.