An optimized second order stochastic learning algorithm for neural network training

Liew, Shan Sung; Khalil-Hani, Mohamed; Bakhteri, Rabia

doi:10.1016/j.neucom.2015.12.076

Cited by 36 publications

(15 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pre-training Performance Layer decomposition [47] Convolutional layers required 2.5× speedup with no loss in accuracy [48] Convolutional layers required 2× speedup with < 1% accuracy drop [52] Whole network required 1.09× reduction in weights & 4.93× speedup in VGG-16 [56] Convolutional layers not required 76% reduction in weights in VGG-11 Pruning [150] Whole network required prune 90% parameters of the convolutional kernels [151] Whole network required prune 13× parameters in VGG-16 [64] Whole network not required 5.1× (CPU) & 3.1× (GPU) speedup in convolutional layers [67] Whole network required 34% inference FLOP reduction in VGG-16 utilize the second order information, which makes it prohibitive in practice for deep large neural networks. Therefore, more emphasis has been put on how to approximate the Hessian matrices, which consists of the second-order derivatives for simplicity [153].…”

Section: Target Layermentioning

confidence: 99%

Recent advances in convolutional neural network acceleration

et al. 2019

View full text Add to dashboard Cite

In recent years, convolutional neural networks (CNNs) have shown great performance in various fields such as image classification, pattern recognition, and multi-media compression. Two of the feature properties, local connectivity and weight sharing, can reduce the number of parameters and increase processing speed during training and inference. However, as the dimension of data becomes higher and the CNN architecture becomes more complicated, the endto-end approach or the combined manner of CNN is computationally intensive, which becomes limitation to CNN's further implementation. Therefore, it is necessary and urgent to implement CNN in a faster way. In this paper, we first summarize the acceleration methods that contribute to but not limited to CNN by reviewing a broad variety of research papers. We propose a taxonomy in terms of three levels, i.e. structure level, algorithm level, and implementation level, for acceleration methods. We also analyze the acceleration methods in terms of CNN architecture compression, algorithm optimization, and hardware-based improvement. At last, we give a discussion on different perspectives of these acceleration and optimization methods within each level. The discussion shows that the methods in each level still have large exploration space. By incorporating such a wide range of disciplines, we expect to provide a comprehensive reference for researchers who are interested in CNN acceleration.

show abstract

Section: Target Layermentioning

confidence: 99%

Recent advances in convolutional neural network acceleration

et al. 2019

View full text Add to dashboard Cite

show abstract

“…After that, the estimated output parameters are compared with the real outputs. For an efficient ANN model, the weights and biases of each layer are updated to estimated outputs with minimum error [20][21][22].…”

Section: Artificial Neural Network and Its Application In Drilling Opmentioning

confidence: 99%

A New Model for Predicting Rate of Penetration Using an Artificial Neural Network

Elkatatny

Al-AbdulJabbar

Abdelgawad

2020

Sensors

View full text Add to dashboard Cite

The drilling rate of penetration (ROP) is defined as the speed of drilling through rock under the bit. ROP is affected by different interconnected factors, which makes it very difficult to infer the mutual effect of each individual parameter. A robust ROP is required to understand the complexity of the drilling process. Therefore, an artificial neural network (ANN) is used to predict ROP and capture the effect of the changes in the drilling parameters. Field data (4525 points) from three vertical onshore wells drilled in the same formation using the same conventional bottom hole assembly were used to train, test, and validate the ANN model. Data from Well A (1528 points) were utilized to train and test the model with a 70/30 data ratio. Data from Well B and Well C were used to test the model. An empirical equation was derived based on the weights and biases of the optimized ANN model and compared with four ROP models using the data set of Well C. The developed ANN model accurately predicted the ROP with a correlation coefficient (R) of 0.94 and an average absolute percentage error (AAPE) of 8.6%. The developed ANN model outperformed four existing models with the lowest AAPE and highest R value.

show abstract

“…This procedure continues until the error is reduced to a certain acceptable limit as shown in Fig. 1 (Liew et al 2016;Naganawa et al 2014;Razi et al 2013).…”

Section: Artificial Neural Network (Ann)mentioning

confidence: 99%

New approach to evaluate the equivalent circulating density (ECD) using artificial intelligence techniques

Abdelgawad

Elzenary

Elkatatny

et al. 2018

J Petrol Explor Prod Technol

View full text Add to dashboard Cite

The equivalent circulation density (ECD) is a very important parameter in drilling high-pressure high-temperature and deepwater wells. ECD is a key parameter during drilling through formations where the margin between the pore pressure and the fracture pressure (FP) is narrow. In these critical formations, the ECD is used to control the formation pressure and prevent kicks. Recent approaches in oilfields to determine ECD depend mainly on using expensive downhole sensors for providing real-time values of ECD. Most of these tools have operational limitations such as high pressure and high temperature which may prevent using these tools in downhole conditions. The objective of this paper is to develop a new approach for predicting ECD using artificial intelligence (AI) techniques from surface drilling parameters [mud weight, drill pipe pressure, and rate of penetration (ROP)]. 2376 data points were used to develop the AI models. The data were collected during the drilling of an 8.5″ vertical hole section. Two AI models were used to estimate the ECD: artificial neural network (ANN) and adaptive neuro-fuzzy inference system (ANFIS). An empirical correlation for ECD was derived from the optimized ANN model by extracting the weights and biases. The developed ANN and ANFIS models were able to calculate ECD with a correlation coefficient (R) of 0.99 and average absolute percentage error of 0.22% for ANN and ANFIS models, respectively. The developed empirical correlation for the ANN model can be used during well design to choose a correct mud weight to safely drill the well based on the expected drilling parameters. It will also minimize the drilling problems related to ECD such as losses or gains especially in critical situations where the margin between the pore and fracture pressure is very narrow. In addition, using this technique will save cost and time by reducing the need for expensive, complicated downhole tools.

show abstract

An optimized second order stochastic learning algorithm for neural network training

Cited by 36 publications

References 31 publications

Recent advances in convolutional neural network acceleration

Recent advances in convolutional neural network acceleration

A New Model for Predicting Rate of Penetration Using an Artificial Neural Network

New approach to evaluate the equivalent circulating density (ECD) using artificial intelligence techniques

Contact Info

Product

Resources

About