HyAdamC: A New Adam-Based Hybrid Optimization Algorithm for Convolution Neural Networks

Abstract

As the performance of devices that conduct large-scale computations has been rapidly improved, various deep learning models have been successfully utilized in various applications. Particularly, convolution neural networks (CNN) have shown remarkable performance in image processing tasks such as image classification and segmentation. Accordingly, more stable and robust optimization methods are required to effectively train them. However, the traditional optimizers used in deep learning still have unsatisfactory training performance for the models with many layers and weights. Accordingly, in this paper, we propose a new Adam-based hybrid optimization method called HyAdamC for training CNNs effectively. HyAdamC uses three new velocity control functions to adjust its search strength carefully in term of initial, short, and long-term velocities. Moreover, HyAdamC utilizes an adaptive coefficient computation method to prevent that a search direction determined by the first momentum is distorted by any outlier gradients. Then, these are combined into one hybrid method. In our experiments, HyAdamC showed not only notable test accuracies but also significantly stable and robust optimization abilities when training various CNN models. Furthermore, we also found that HyAdamC could be applied into not only image classification and image segmentation tasks.

Development of an Artificial Neural Network for the Detection of Supporting Hindlimb Lameness: A Pilot Study in Working Dogs

“…The gradient descent algorithm is mostly used for these purposes [ 21 ]. In our study, we specifically used the Adam algorithm , which is one of the most widely used optimization algorithms due to advantages such as high efficiency and low resource consumption [ 22 ].…”

Preliminary Classification of Selected Farmland Habitats in Ireland Using Deep Neural Networks

“…Each model was trained with different optimizers by varying the learning rates and the best-fit hyperparameters were chosen as the ones with the highest validation accuracy. Based on previous work in image classification, the four most suitable optimizers, namely, Adam, RMSProp, SGD, and Nadam were chosen for tuning each model [ 70 , 71 , 72 ]. The learning rate was varied from 10 −2 to 10 −5 to determine the best fit.…”

Socio-Economic Factors Affecting Member’s Satisfaction towards National Health Insurance: An Evidence from the Philippines

“…Two AFs are considered for the output layer: Sigmoid [ 79 , 81 , 88 , 89 ] and Softmax [ 90 , 91 ]. Lastly, three optimizers are used, namely, Adam [ 92 , 93 , 94 ], RMSProp [ 90 , 95 , 96 ], and SGD [ 97 , 98 ].…”