SinLU: Sinu-Sigmoidal Linear Unit

Paul, Ashis Kumar; Bandyopadhyay, Rajarshi; Lee, Ji Eun; Geem, Zong Woo; Sarkar, Ram

doi:10.3390/math10030337

Cited by 13 publications

(6 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 2022, scholars including Paul Ashis, Bandyopadhyay Rajarshi, Yoon Jin, Geem Zong Woo, and Sarkar Ram introduced the sinusoidal sigmoid linear unit (SinLU) activation function [15]. The SinLU activation function is designed with two trainable parameters: one controlling the amplitude of the sinusoidal function and the other regulating the frequency of the sine wave.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Enhancing Deep Learning Models for Image Classification using Hybrid Activation Functions

Zhang,

Li,

Yang

et al. 2023

Preprint

View full text Add to dashboard Cite

In the era of big data, efficient data processing has become a crucial issue for scientific development. Image classification, as one of the core tasks in the field of computer vision, holds great significance for achieving automated and intelligent applications. Nonlinear activation functions play a crucial role in neural networks, as they can introduce nonlinear properties and improve the representation and learning ability of the model. Therefore, it is essential to investigate the performance of different nonlinear activation functions on image classification tasks to optimize model performance and improve data processing efficiency. This paper is based on three nonlinear activation functions, namely, the cosine linear unit (CosLU), derivative exponential linear unit (DELU), and rectified linear unit with nonnegative slope (ReLUN), proposed by E. Pishchik in 2023, to study their performance on image classification tasks. We selected two datasets, CIFAR-10 and CIFAR-100, and employed these three activation functions to train five progressively deepening network models. By comparing them with the ReLU activation function and between the two datasets, we expanded the number of classes in the dataset to provide a more comprehensive evaluation of these activation functions. The experimental results demonstrate that when trained on the CIFAR-10 dataset, the cosine linear unit (CosLU) activation function outperforms ReLU, while the derivative exponential linear unit (DELU) activation function exhibits poor performance, and the rectified linear unit with nonnegative slope (ReLUN) activation function performs similarly to ReLU. However, when trained on the CIFAR-100 dataset, the effectiveness of these activation functions significantly decreases. Additionally, we observed that activation functions with trainable parameters tend to exhibit an overall performance trend that improves as the model size increases. Furthermore, we identified a characteristic shared by most activation functions with trainable parameters, indicating that the larger the model is, the better the overall performance trend may become.

show abstract

Section: Resultsmentioning

confidence: 99%

“…The CosLU activation function is an enhancement derived from the sinusoidal Sigmoidal Linear Unit (SinLU) activation function proposed by Paul Ashis et al [23]. In contrast to the SinLU activation function, which utilizes the sine function, the CosLU activation function employs the cosine function as the periodic function.…”

Section: Coslumentioning

confidence: 99%

Enhancing Deep Learning Models for Image Classification using Hybrid Activation Functions

Zhang,

Li,

Yang

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The Mish activation function, introduced by Misra (2019), provided a smooth, nonlinear alternative that excels in tasks like image classification, albeit with higher computational demands, as noted by Zhang et al (2021). The recent SinLU activation function by Paul et al (2022) further expanded the landscape by incorporating two trainable parameters and leveraging the periodicity of the sine function to introduce novel dynamics into neural network training.…”

Section: Related Workmentioning

confidence: 99%

Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

Zhang,

Li,

et al. 2024

IJWIS

View full text Add to dashboard Cite

Purpose The purpose of this study is to explore the potential of trainable activation functions to enhance the performance of deep neural networks, specifically ResNet architectures, in the task of image classification. By introducing activation functions that adapt during training, the authors aim to determine whether such flexibility can lead to improved learning outcomes and generalization capabilities compared to static activation functions like ReLU. This research seeks to provide insights into how dynamic nonlinearities might influence deep learning models' efficiency and accuracy in handling complex image data sets. Design/methodology/approach This research integrates three novel trainable activation functions – CosLU, DELU and ReLUN – into various ResNet-n architectures, where “n” denotes the number of convolutional layers. Using CIFAR-10 and CIFAR-100 data sets, the authors conducted a comparative study to assess the impact of these functions on image classification accuracy. The approach included modifying the traditional ResNet models by replacing their static activation functions with the trainable variants, allowing for dynamic adaptation during training. The performance was evaluated based on accuracy metrics and loss profiles across different network depths. Findings The findings indicate that trainable activation functions, particularly CosLU, can significantly enhance the performance of deep learning models, outperforming the traditional ReLU in deeper network configurations on the CIFAR-10 data set. CosLU showed the highest improvement in accuracy, whereas DELU and ReLUN offered varying levels of performance enhancements. These functions also demonstrated potential in reducing overfitting and improving model generalization across more complex data sets like CIFAR-100, suggesting that the adaptability of activation functions plays a crucial role in the training dynamics of deep neural networks. Originality/value This study contributes to the field of deep learning by introducing and evaluating the impact of three novel trainable activation functions within widely used ResNet architectures. Unlike previous works that primarily focused on static activation functions, this research demonstrates that incorporating trainable nonlinearities can lead to significant improvements in model performance and adaptability. The introduction of CosLU, DELU and ReLUN provides a new pathway for enhancing the flexibility and efficiency of neural networks, potentially setting a new standard for future deep learning applications in image classification and beyond.

show abstract

“…The Backbone component is primarily based on the CBS, [24] C3, [25] and SPP [26] structures. The CBS structure consists of a 2D convolutional layer Conv2d, [27] a batch normalization procedure BatchNorm2d, [28] and an activation function SiLU [29] in sequence. The C3 structure is split into two branches: the upper branch passes through the standard convolution and the Bottleneck module, while the lower branch joins the standard convolution, is then spliced with the higher branch, and finally connects to another standard convolution.…”

Section: Yolov5 Algorithmmentioning

confidence: 99%

Improving the Accuracy of Robot Collecting Organisms in Marine Environment Based on Yolov5 Improvement

Wang,

Yuan,

Wang

et al. 2023

Advcd Theory and Sims

View full text Add to dashboard Cite

In this paper, an improved YOLOv5 multiscale marine organism target detection algorithm (YOLOv5‐Mult) is proposed to address the insufficient feature extraction ability of small targets, low detection accuracy, and high catching error of existing models in complex environments. First, real frame clustering is performed using the Kmeans++ method. Second, the BiFPN network module is adopted in lieu of the PANet network module to enhance the feature fusion ability. Next, the multilayer semantic fusion module RBC (RepBlock CSP) replaces the C3 module before the SPP layer of the Backbone network and the C3 module in the Neck layer to enrich the image semantic information. Finally, the multiscale feature fusion module MC (Mult Conv) replaces the last C3 module in the Backbone network to mitigate the semantic gap between different feature channel layers. Experimental results demonstrate that the improved algorithm attains a mAP value of 71.18%, which is 5.22% higher than that of the original YOLOv5 algorithm, providing accurate identification and fishing for underwater robots.

show abstract

SinLU: Sinu-Sigmoidal Linear Unit

Cited by 13 publications

References 25 publications

Enhancing Deep Learning Models for Image Classification using Hybrid Activation Functions

Enhancing Deep Learning Models for Image Classification using Hybrid Activation Functions

Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

Improving the Accuracy of Robot Collecting Organisms in Marine Environment Based on Yolov5 Improvement

Contact Info

Product

Resources

About