Practical Convex Formulations of One-hidden-layer Neural Network Adversarial Training

Yatong, Bai,; Tanmay, Gautam,; Gai, Yu; Sojoudi, Somayeh

doi:10.23919/acc53348.2022.9867244

Cited by 2 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apicella et al (2021) provided a comprehensive survey on modern trainable activation functions, highlighting their importance in enhancing learning capabilities. The Gaussian error function-based activation function introduced by Chen and Pock (2016) exemplifies the trend toward smoother alternatives to ReLU, despite the computational trade-offs as reported by Bai et al (2023). Sonoda and Murata (2017) advanced this line of inquiry by adapting the Fourier series and Gaussian cumulative distribution function to devise activation functions for particular architectures.…”

Section: Related Workmentioning

confidence: 93%

Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

Zhang,

Li,

et al. 2024

IJWIS

View full text Add to dashboard Cite

Purpose The purpose of this study is to explore the potential of trainable activation functions to enhance the performance of deep neural networks, specifically ResNet architectures, in the task of image classification. By introducing activation functions that adapt during training, the authors aim to determine whether such flexibility can lead to improved learning outcomes and generalization capabilities compared to static activation functions like ReLU. This research seeks to provide insights into how dynamic nonlinearities might influence deep learning models' efficiency and accuracy in handling complex image data sets. Design/methodology/approach This research integrates three novel trainable activation functions – CosLU, DELU and ReLUN – into various ResNet-n architectures, where “n” denotes the number of convolutional layers. Using CIFAR-10 and CIFAR-100 data sets, the authors conducted a comparative study to assess the impact of these functions on image classification accuracy. The approach included modifying the traditional ResNet models by replacing their static activation functions with the trainable variants, allowing for dynamic adaptation during training. The performance was evaluated based on accuracy metrics and loss profiles across different network depths. Findings The findings indicate that trainable activation functions, particularly CosLU, can significantly enhance the performance of deep learning models, outperforming the traditional ReLU in deeper network configurations on the CIFAR-10 data set. CosLU showed the highest improvement in accuracy, whereas DELU and ReLUN offered varying levels of performance enhancements. These functions also demonstrated potential in reducing overfitting and improving model generalization across more complex data sets like CIFAR-100, suggesting that the adaptability of activation functions plays a crucial role in the training dynamics of deep neural networks. Originality/value This study contributes to the field of deep learning by introducing and evaluating the impact of three novel trainable activation functions within widely used ResNet architectures. Unlike previous works that primarily focused on static activation functions, this research demonstrates that incorporating trainable nonlinearities can lead to significant improvements in model performance and adaptability. The introduction of CosLU, DELU and ReLUN provides a new pathway for enhancing the flexibility and efficiency of neural networks, potentially setting a new standard for future deep learning applications in image classification and beyond.

show abstract

Section: Related Workmentioning

confidence: 93%

Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

Zhang,

Li,

et al. 2024

IJWIS

View full text Add to dashboard Cite

show abstract

“…The vulnerability of neural networks to adversarial attacks has been observed in various applications, such as computer vision [25,44] and control systems [31]. In response, "adversarial training" [12,13,25,36,62] has been studied to alleviate the susceptibility. Adversarial training builds robust neural networks by training on adversarial examples.…”

Section: Introductionmentioning

confidence: 99%

Improving the Accuracy-Robustness Trade-off of Classifiers via Adaptive Smoothing

Yatong¹,

Anderson²,

Kim³

et al. 2023

Preprint

View full text Add to dashboard Cite

While it is shown in the literature that simultaneously accurate and robust classifiers exist for common datasets, previous methods that improve the adversarial robustness of classifiers often manifest an accuracy-robustness trade-off. We build upon recent advancements in data-driven "locally biased smoothing" to develop classifiers that treat benign and adversarial test data differently. Specifically, we tailor the smoothing operation to the usage of a robust neural network as the source of robustness. We then extend the smoothing procedure to the multi-class setting and adapt an adversarial input detector into a policy network. The policy adaptively adjusts the mixture of the robust base classifier and a standard network, where the standard network is optimized for clean accuracy and is not robust in general. We provide theoretical analyses to motivate the use of the adaptive smoothing procedure, certify the robustness of the smoothed classifier under realistic assumptions, and justify the introduction of the policy network. We use various attack methods, including AutoAttack and adaptive attack, to empirically verify that the smoothed model noticeably improves the accuracy-robustness trade-off. On the CIFAR-100 dataset, our method simultaneously achieves an 80.09% clean accuracy and a 32.94% AutoAttacked accuracy. The code that implements adaptive smoothing is available at https://github.com/Bai-YT/AdaptiveSmoothing.

show abstract

Practical Convex Formulations of One-hidden-layer Neural Network Adversarial Training

Cited by 2 publications

References 6 publications

Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

Improving the Accuracy-Robustness Trade-off of Classifiers via Adaptive Smoothing

Contact Info

Product

Resources

About