Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems

Liew, Shan Sung; Khalil-Hani, Mohamed; Bakhteri, Rabia

doi:10.1016/j.neucom.2016.08.037

Cited by 143 publications

(66 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Implementation of ReLU is possible with a thresholding of an activation map at zero. However, learning with ReLU in gradient descent‐based techniques is not possible because all gradient values will be zero when the activation values are zero . Therefore, in this work, LeakyReLU that is defined by

F_{LeakyReLU} = ({, \begin{array}{c} x & italicif x > 0 \\ 0.01 x & italicelse \end{array})

has been implemented to provide learning efficiently, even if the activation values are zero.…”

Section: Proposed Methodsmentioning

confidence: 99%

“…However, learning with ReLU in gradient descentbased techniques is not possible because all gradient values will be zero when the activation values are zero. 24 Therefore, in this work, LeakyReLU 24 that is defined by…”

Section: Activation Function In the 3d Cnn Architecturementioning

confidence: 99%

“…However, deep learning‐based network models have many algorithmic components to be designed, such as architecture design (number of layers, filter width and sizes, and regularization layers), activation functions, and optimization functions, which considerably affect the performance of the method. For instance, learning with Rectified Linear Unit (ReLU) can reduce performance in gradient descent operations because all gradient values will be zero when the activation values are zero . Therefore, in the literature, these approaches have been established after many ad‐hoc and trial‐error engineering designs.…”

Section: Introductionmentioning

confidence: 99%

“…For instance, learning with Rectified Linear Unit (ReLU) can reduce performance in gradient descent operations because all gradient values will be zero when the activation values are zero. 24 Therefore, in the literature, these approaches have been established after many ad-hoc and trial-error engineering designs. Also, different data sets obtained from different imaging parameters and techniques have been used.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Diagnosis of Alzheimer's disease with Sobolev gradient‐based optimization and 3D convolutional neural network

Göçeri

2019

Numer Methods Biomed Eng

View full text Add to dashboard Cite

Alzheimer's disease is a neuropsychiatric, progressive, also an irreversible disease. There is not an effective cure for the disease. However, early diagnosis has an important role for treatment planning to delay its progression since the treatments have the most impact at the early stage of the disease. Neuroimages obtained by different imaging techniques (for example, diffusion tensor‐based and magnetic resonance‐based imaging) provide powerful information and help to diagnose the disease. In this work, a deeply supervised and robust method has been developed using three dimensional features to provide objective and accurate diagnosis from magnetic resonance images. The main contributions are (a) a new three dimensional convolutional neural network topology; (b) a new Sobolev gradient‐based optimization with weight values for each decision parameters; (c) application of the proposed topology and optimizer to diagnose Alzheimer's disease; (d) comparisons of the results obtained from the recent techniques that have been implemented for Alzheimer's disease diagnosis. Experimental results and quantitative evaluations indicated that the proposed network model is able to achieve to extract desired features from images and provides automated diagnosis with 98.06% accuracy.

show abstract

F_{LeakyReLU} = ({, \begin{array}{c} x & italicif x > 0 \\ 0.01 x & italicelse \end{array})

has been implemented to provide learning efficiently, even if the activation values are zero.…”

Section: Proposed Methodsmentioning

confidence: 99%

Section: Activation Function In the 3d Cnn Architecturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Diagnosis of Alzheimer's disease with Sobolev gradient‐based optimization and 3D convolutional neural network

Göçeri

2019

Numer Methods Biomed Eng

View full text Add to dashboard Cite

show abstract

“…But in model parameter training, weight matrices and bias vectors are updated using an error back-propagation algorithm whereas activation function is not. So the change of activation function is important for a neural network, which can speed up model training [17], enhance stability [18]. In this paper, we adopt the RNN model and modify its activation function to do NER task.…”

Section: Related Workmentioning

confidence: 99%

Co-training an Improved Recurrent Neural Network with Probability Statistic Models for Named Entity Recognition

Sun

Xie

et al. 2017

Database Systems for Advanced Applications

View full text Add to dashboard Cite

Abstract. Named Entity Recognition (NER) is a subtask of information extraction in Natural Language Processing (NLP) field and thus being wildly studied. Currently Recurrent Neural Network (RNN) has become a popular way to do NER task, but it needs a lot of train data. The lack of labeled train data is one of the hard problems and traditional co-training strategy is a way to alleviate it. In this paper, we consider this situation and focus on doing NER with co-training using RNN and two probability statistic models i.e. Hidden Markov Model (HMM) and Conditional Random Field (CRF). We proposed a modified RNN model by redefining its activation function. Compared to traditional sigmoid function, our new function avoids saturation to some degree and makes its output scope very close to [0, 1], thus improving recognition accuracy. Our experiments are conducted ATIS benchmark. First, supervised learning using those models are compared when using different train data size. The experimental results show that it is not necessary to use whole data, even small part of train data can also get good performance. Then, we compare the results of our modified RNN with original RNN. 0.5% improvement is obtained. Last, we compare the co-training results. HMM and CRF get higher improvement than RNN after co-training. Moreover, using our modified RNN in co-training, their performances are improved further.

show abstract

Machine‐Learning‐Assisted Determination of the Global Zero‐Temperature Phase Diagram of Materials

et al. 2023

View full text Add to dashboard Cite

Crystal‐graph attention neural networks have emerged recently as remarkable tools for the prediction of thermodynamic stability. The efficacy of their learning capabilities and their reliability is however subject to the quantity and quality of the data they are fed. Previous networks exhibit strong biases due to the inhomogeneity of the training data. Here a high‐quality dataset is engineered to provide a better balance across chemical and crystal‐symmetry space. Crystal‐graph neural networks trained with this dataset show unprecedented generalization accuracy. Such networks are applied to perform machine‐learning‐assisted high‐throughput searches of stable materials, spanning 1 billion candidates. In this way, the number of vertices of the global T = 0 K phase diagram is increased by 30% and find more than ≈150 000 compounds with a distance to the convex hull of stability of less than 50 meV atom−1. The discovered materials are then accessed for applications, identifying compounds with extreme values of a few properties, such as superconductivity, superhardness, and giant gap‐deformation potentials.

show abstract

Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems

Cited by 143 publications

References 39 publications

Diagnosis of Alzheimer's disease with Sobolev gradient‐based optimization and 3D convolutional neural network

Diagnosis of Alzheimer's disease with Sobolev gradient‐based optimization and 3D convolutional neural network

Co-training an Improved Recurrent Neural Network with Probability Statistic Models for Named Entity Recognition

Machine‐Learning‐Assisted Determination of the Global Zero‐Temperature Phase Diagram of Materials

Contact Info

Product

Resources

About