Enhancing Deep Learning Models for Image Classification using Hybrid Activation Functions

Abstract

In the era of big data, efficient data processing has become a crucial issue for scientific development. Image classification, as one of the core tasks in the field of computer vision, holds great significance for achieving automated and intelligent applications. Nonlinear activation functions play a crucial role in neural networks, as they can introduce nonlinear properties and improve the representation and learning ability of the model. Therefore, it is essential to investigate the performance of different nonlinear activation functions on image classification tasks to optimize model performance and improve data processing efficiency. This paper is based on three nonlinear activation functions, namely, the cosine linear unit (CosLU), derivative exponential linear unit (DELU), and rectified linear unit with nonnegative slope (ReLUN), proposed by E. Pishchik in 2023, to study their performance on image classification tasks. We selected two datasets, CIFAR-10 and CIFAR-100, and employed these three activation functions to train five progressively deepening network models. By comparing them with the ReLU activation function and between the two datasets, we expanded the number of classes in the dataset to provide a more comprehensive evaluation of these activation functions. The experimental results demonstrate that when trained on the CIFAR-10 dataset, the cosine linear unit (CosLU) activation function outperforms ReLU, while the derivative exponential linear unit (DELU) activation function exhibits poor performance, and the rectified linear unit with nonnegative slope (ReLUN) activation function performs similarly to ReLU. However, when trained on the CIFAR-100 dataset, the effectiveness of these activation functions significantly decreases. Additionally, we observed that activation functions with trainable parameters tend to exhibit an overall performance trend that improves as the model size increases. Furthermore, we identified a characteristic shared by most activation functions with trainable parameters, indicating that the larger the model is, the better the overall performance trend may become.