On the Variance of the Adaptive Learning Rate and Beyond

Liu, Liyuan; Jiang, Haoming; He, Pengcheng; Chen, Weizhu; Liu, Xiaodong; Gao, Jianfeng

doi:10.48550/arxiv.1908.03265

Cited by 309 publications

(335 citation statements)

References 15 publications

Supporting

Mentioning

332

Contrasting

Order By: Relevance

“…The above results can be extended to include the presence of a state-dependent,time-varying, learning rate [Zeiler, 2012, Kingma and Ba, 2014, Goodfellow et al, 2016, Liu et al, 2019.…”

Section: Adaptive Learning Ratesmentioning

confidence: 99%

Generalization in Supervised Learning Through Riemannian Contraction

Kozachkov¹,

Wensing²,

Slotine³

2022

Preprint

View full text Add to dashboard Cite

We prove that Riemannian contraction in a supervised learning setting implies generalization. Specifically, we show that if an optimizer is contracting in some Riemannian metric with rate λ > 0, it is uniformly algorithmically stable with rate O(1/λn), where n is the number of labelled examples in the training set. The results hold for stochastic and deterministic optimization, in both continuous and discrete-time, for convex and non-convex loss surfaces. The associated generalization bounds reduce to well-known results in the particular case of gradient descent over convex or strongly convex loss surfaces. They can be shown to be optimal in certain linear settings, such as kernel ridge regression under gradient flow.

show abstract

“…The above results can be extended to include the presence of a state-dependent,time-varying, learning rate [Zeiler, 2012, Kingma and Ba, 2014, Goodfellow et al, 2016, Liu et al, 2019.…”

Section: Adaptive Learning Ratesmentioning

confidence: 99%

Generalization in Supervised Learning Through Riemannian Contraction

Kozachkov¹,

Wensing²,

Slotine³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…which is used as the target for optimisation [9]. Optimisation was performed using the Rectified ADAM (RADAM) algorithm [30,31] combined with early stopping on a separate validation data split. The parameters of the Rectified ADAM algorithm were varied depending on the input-output subset based on experiments with the validation set.…”

Section: Neural Network Optimisationmentioning

confidence: 99%

In-flight Novelty Detection with Convolutional Neural Networks

Hartwell¹,

Montana²,

Jacobs³

et al. 2021

Preprint

View full text Add to dashboard Cite

Gas turbine engines are complex machines that typically generate a vast amount of data, and require careful monitoring to allow for cost-effective preventative maintenance. In aerospace applications, returning all measured data to ground is prohibitively expensive, often causing useful, high value, data to be discarded. The ability to detect, prioritise, and return useful data in real-time is therefore vital. This paper proposes that system output measurements, described by a convolutional neural network model of normality, are prioritised in real-time for the attention of preventative maintenance decision makers.Due to the complexity of gas turbine engine timevarying behaviours, deriving accurate physical models is difficult, and often leads to models with low prediction accuracy and incompatibility with real-time execution. Data-driven modelling is a desirable alternative producing high accuracy, asset specific models without the need for derivation from first principles.We present a data-driven system for online detection and prioritisation of anomalous data. Biased data assessment deriving from novel operating conditions is avoided by uncertainty management integrated into the deep neural predictive model. Testing is performed on real and synthetic data, showing sensitivity to both real and synthetic faults. The system is capable of running in real-time on low-power embedded hardware and is currently in deployment on the Rolls-Royce Pearl 15 engine flight trials.

show abstract

“…For E 2 , we modify the style block of original backbone to output 512×8×8 tensors instead of 512 vectors. In Phase I, we follow the previous encoder-based methods [3,38,45] and use the Ranger optimizer, which combines the Lookahead [55] and the Rectified Adam [31] optimizer, for training. In Phase II, we use Adam [28] with standard settings, which we found to make the training of the hypernetworks converges faster.…”

Section: Experiments 41 Experimental Settingsmentioning

confidence: 99%

HyperInverter: Improving StyleGAN Inversion via Hypernetwork

Dinh¹,

Tran²,

Nguyen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Real-world image manipulation has achieved fantastic progress in recent years as a result of the exploration and utilization of GAN latent spaces. GAN inversion is the first step in this pipeline, which aims to map the real image to the latent code faithfully. Unfortunately, the majority of existing GAN inversion methods fail to meet at least one of the three requirements listed below: high reconstruction quality, editability, and fast inference. We present a novel two-phase strategy in this research that fits all requirements at the same time. In the first phase, we train an encoder to map the input image to StyleGAN2 W-space, which was proven to have excellent editability but lower reconstruction quality. In the second phase, we supplement the reconstruction ability in the initial phase by leveraging a series of hypernetworks to recover the missing information during inversion. These two steps complement each other to yield high reconstruction quality thanks to the hypernetwork branch and excellent editability due to the inversion done in the W-space. Our method is entirely encoder-based, resulting in extremely fast inference. Extensive experiments on two challenging datasets demonstrate the superiority of our method. 1

show abstract

On the Variance of the Adaptive Learning Rate and Beyond

Cited by 309 publications

References 15 publications

Generalization in Supervised Learning Through Riemannian Contraction

Generalization in Supervised Learning Through Riemannian Contraction

In-flight Novelty Detection with Convolutional Neural Networks

HyperInverter: Improving StyleGAN Inversion via Hypernetwork

Contact Info

Product

Resources

About