Nesterov's accelerated gradient and momentum as approximations to regularised update descent

Botev, Aleksandar; Lever, Guy; Barber, David

doi:10.1109/ijcnn.2017.7966082

Cited by 96 publications

(48 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dropout regularization ( Srivastava et al, 2014 ) with a dropout ratio of 0.5 is applied to outputs of the first fully connected layer. The model is trained by optimizing the multinomial logistic regression objective using stochastic gradient descent (SGD) ( LeCun, Bengio & Hinton, 2015 ) and Nesterov’s momentum ( Botev, Lever & Barber, 2017 ). The customized model is optimized for hyper-parameters by a randomized grid search method ( Bergstra & Bengio, 2012 ).…”

Section: Methodsmentioning

confidence: 99%

Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images

Rajaraman

Antani

Poostchi

et al. 2018

PeerJ

411

265

View full text Add to dashboard Cite

Malaria is a blood disease caused by the Plasmodium parasites transmitted through the bite of female Anopheles mosquito. Microscopists commonly examine thick and thin blood smears to diagnose disease and compute parasitemia. However, their accuracy depends on smear quality and expertise in classifying and counting parasitized and uninfected cells. Such an examination could be arduous for large-scale diagnoses resulting in poor quality. State-of-the-art image-analysis based computer-aided diagnosis (CADx) methods using machine learning (ML) techniques, applied to microscopic images of the smears using hand-engineered features demand expertise in analyzing morphological, textural, and positional variations of the region of interest (ROI). In contrast, Convolutional Neural Networks (CNN), a class of deep learning (DL) models promise highly scalable and superior results with end-to-end feature extraction and classification. Automated malaria screening using DL techniques could, therefore, serve as an effective diagnostic aid. In this study, we evaluate the performance of pre-trained CNN based DL models as feature extractors toward classifying parasitized and uninfected cells to aid in improved disease screening. We experimentally determine the optimal model layers for feature extraction from the underlying data. Statistical validation of the results demonstrates the use of pre-trained CNNs as a promising tool for feature extraction for this purpose.

show abstract

Section: Methodsmentioning

confidence: 99%

Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images

Rajaraman

Antani

Poostchi

et al. 2018

PeerJ

411

265

View full text Add to dashboard Cite

show abstract

“…Each Training-ValueNet is a MLP regression network with a single hidden layer of 1024 units. Training is carried out using minibatch SGD with a batch size of 32 and 0.9 Nesterov momentum [25]. We also use dropout [22] after the hidden layer at a rate of 0.7.…”

Section: Monte-carlo Estimationmentioning

confidence: 99%

Training-ValueNet: Data Driven Label Noise Cleaning on Weakly-Supervised Web Images

Smyth

Kangin

Pugeault

2019

2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)

View full text Add to dashboard Cite

Manually labelling new datasets for image classification remains expensive and time-consuming. A promising alternative is to utilize the abundance of images on the web for which search queries or surrounding text offers a natural source of weak supervision. Unfortunately the label noise in these datasets has limited their use in practice. Several methods have been proposed for performing unsupervised label noise cleaning, the majority of which use outlier detection to identify and remove mislabeled images. In this paper, we argue that outlier detection is an inherently unsuitable approach for this task due to major flaws in the assumptions it makes about the distribution of mislabeled images. We propose an alternative approach which makes no such assumptions. Rather than looking for outliers, we observe that mislabeled images can be identified by the detrimental impact they have on the performance of an image classifier. We introduce training-value as an objective measure of the contribution each training example makes to the validation loss. We then present the training-value approximation network (Training-ValueNet) which learns a mapping between each image and its training-value. We demonstrate that by simply discarding images with a negative training-value, Training-ValueNet is able to significantly improve classification performance on a held-out test set, outperforming the state of the art in outlier detection by a large margin.

show abstract

“…The momentum term (Qian, 1999) of SGD helps in accelerating the process by allowing the SGD to navigate better in ravines. However, although the momentum term has proved extremely useful, there has been an improvement on it which is known as Nesterov Accelerated Gradient (NAG) (Botev et al, 2017). This allows the calculation of the gradient not based on the current parameters but based on the future position of the parameters.…”

Section: System Designmentioning

confidence: 99%

Concerns on the issue of defence expenditure in the post-crisis Greece

Katsaitis

Kondylis

Zombanakis

2019

Security and Defence Quarterly

View full text Add to dashboard Cite

The paper aims to tackle a controversial issue, namely the anticipated developments regarding defence expenditure once the Greek economy returns to growth. Such a comeback is expected to occur following a prolonged recessionary period during which defence spending cuts were a top priority, as recommended by the IMF, the ECB and the EC, members of the so-called "Troika". The paper uses both conventional econometrics as well as neural networks to consider and evaluate the hierarchy's ordering of the determinants used in such a demand for defence expenditure based on their explanatory power. While the role of property resources is certainly pronounced, as expected, human resources variables also seem to be able to explain defence spending developments, especially in the recent past. A forecasting investigation based on this background points to a number of interesting conclusions on the anticipated developments concerning defence spending in the future as well as on the determinants of such developments which might represent a threat to NATO cohesion.1 Popular term widely used in Greece, Cyprus, Ireland, Portugal and Spain to refer to the presence of the International Monetary Fund, the European Central Bank and the European Commission in these countries since 2010 and the economic policy measures that these institutions have proposed and monitored in order to deal with the economic problems arising in each case. 2 In fact there is more to this issue than what meets the eye (Ragies, 2017): Indeed, during the recent NATO summit in July, it has been pointed out that only five allies (US, UK, Poland, Greece and Estonia) contribute 2% or more of their GDP to defence. The fact remains, however, that regarding Greece, roughly 70% of its defence spending represents inelastic spending on salaries, wages and pensions of military and civilian personnel and only about 25% to equipment and infrastructure spending, which includes contributions to the alliance such as the NMIOTC (NATO Maritime Interdiction Operational Training Centre) in Crete.

show abstract

Nesterov's accelerated gradient and momentum as approximations to regularised update descent

Cited by 96 publications

References 3 publications

Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images

Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images

Training-ValueNet: Data Driven Label Noise Cleaning on Weakly-Supervised Web Images

Concerns on the issue of defence expenditure in the post-crisis Greece

Contact Info

Product

Resources

About