On the Impact of the Activation Function on Deep Neural Networks Training

Hayou, Soufiane; Doucet, Arnaud; Rousseau, Judith

doi:10.48550/arxiv.1902.06853

Cited by 12 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The initialization of weights is a significant step that can impact the learning Fig. 3 The bidirectional RNN structure over the time process and, ultimately, the model's convergence (Hayou et al, 2019).…”

Section: Model Training Phasementioning

confidence: 99%

Building a neural speech recognizer for quranic recitations

Al-Issa

Al‐Ayyoub

Al-Khaleel

et al. 2022

Int J Speech Technol

View full text Add to dashboard Cite

This work is an effort towards building Neural Speech Recognizers system for Quranic recitations that can be effectively used by anyone regardless of their gender and age. Despite having a lot of recitations available online, most of them are recorded by professional male adult reciters, which means that an ASR system trained on such datasets would not work for female/ child reciters. We address this gap by adopting a benchmark dataset of audio records of Quranic recitations that consists of recitations by both genders from different ages. Using this dataset, we build several speaker-independent NSR systems based on the DeepSpeech model and use word error rate (WER) for evaluating them. The goal is to show how an NSR system trained and tuned on a dataset of a certain gender would perform on a test set from the other gender. Unfortunately, the number of female recitations in our dataset is rather small while the number of male recitations is much larger. In the first set of experiments, we avoid the imbalance issue between the two genders and down-sample the male part to match the female part. For this small subset of our dataset, the results are interesting with 0.968 WER when the system is trained on male recitations and tested on female recitations. The same system gives 0.406 WER when tested on male recitations. On the other hand, training the system on female recitations and testing it on male recitation gives 0.966 WER while testing it on female recitations gives 0.608 WER.

show abstract

Section: Model Training Phasementioning

confidence: 99%

Building a neural speech recognizer for quranic recitations

Al-Issa

Al‐Ayyoub

Al-Khaleel

et al. 2022

Int J Speech Technol

View full text Add to dashboard Cite

show abstract

“…The trainability of a neural network indicates how effective it can be optimized using gradient descent (Burkholz & Dubatovka, 2019;Hayou et al, 2019;Shin & Karniadakis, 2020). Although some heavy networks can theoretically represent complex functions, they not necessarily can be effectively trained by gradient descent.…”

Section: Trainability By Condition Number Of Ntkmentioning

confidence: 99%

Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective

Chen¹,

Gong²,

Wang³

2021

Preprint

View full text Add to dashboard Cite

Neural Architecture Search (NAS) has been explosively studied to automate the discovery of top-performer neural networks. Current works require heavy training of supernet or intensive architecture evaluations, thus suffering from heavy resource consumption and often incurring search bias due to truncated training or approximations. Can we select the best neural architectures without involving any training and eliminate a drastic portion of the search cost? We provide an affirmative answer, by proposing a novel framework called training-free neural architecture search (TE-NAS). TE-NAS ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space. Both are motivated by recent theory advances in deep networks and can be computed without any training and any label. We show that: (1) these two measurements imply the trainability and expressivity of a neural network; (2) they strongly correlate with the network's test accuracy. Further on, we design a pruning-based NAS mechanism to achieve a more flexible and superior trade-off between the trainability and expressivity during the search. In NAS-Bench-201 and DARTS search spaces, TE-NAS completes high-quality search but only costs 0.5 and 4 GPU hours with one 1080Ti on CIFAR-10 and ImageNet, respectively. We hope our work inspires more attempts in bridging the theoretical findings of deep networks and practical impacts in real NAS applications. Code is available at: https://github.com/VITA-Group/TENAS.

show abstract

“…), even the former could equip a larger number of parameters. The trainability of a neural network studies how effective it can be optimized by gradient descent [44], [45], [46].…”

Section: Trainabilitymentioning

confidence: 99%

Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics

Chen¹,

Gong²,

Wei³

et al. 2021

Preprint

View full text Add to dashboard Cite

This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works [1], [2], [3] start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators are not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling "TEG" characteristics of searched networks -Trainability, Expressivity, Generalization -all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 2.3× reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of "TEG" characteristics, observing that while a good local minimum is easier to find on NAS-Bench-201 given its simple topology, balancing "TEG" characteristics is much harder on the DARTS search space due to its complex landscape geometry. Our code is available at https://github.com/VITA-Group/TEGNAS.

show abstract

On the Impact of the Activation Function on Deep Neural Networks Training

Cited by 12 publications

References 0 publications

Building a neural speech recognizer for quranic recitations

Building a neural speech recognizer for quranic recitations

Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective

Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics

Contact Info

Product

Resources

About