A New Nonmonotone Adaptive Retrospective Trust Region Method for Unconstrained Optimization Problems

Tarzanagh, Davoud Ataee; Peyghami, M. Reza; Bastin, Fabian

doi:10.1007/s10957-015-0790-0

Cited by 2 publications

(1 citation statement)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Appropriately choosing the learning rate that scales coordinates of the gradient and the way of updating them are critical issues that impact the performance of first [27,28,29] and second order optimization procedures [30,31,32]. Indeed, an adaptive learning rate is advantageous, which led to the development of a family of widely-used methods including ADAGRAD [27], ADADELTA [33], RMSPROP [29], ADAM [28] and AMSGRAD [34].…”

Section: Introductionmentioning

confidence: 99%

DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimization

Parvin¹,

Tarzanagh²,

Michailidis³

2019

Preprint

View full text Add to dashboard Cite

Adaptive gradient-based optimization methods such as ADAGRAD, RMSPROP, and ADAM are widely used in solving large-scale machine learning problems including deep learning. A number of schemes have been proposed in the literature aiming at parallelizing them, based on communications of peripheral nodes with a central node, but incur high communications cost. To address this issue, we develop a novel consensus-based distributed adaptive moment estimation method (DADAM) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation. The method is particularly useful, since it can accommodate settings where access to local data is allowed. Further, as established theoretically in this work, it can outperform centralized adaptive algorithms, for certain classes of loss functions used in applications. We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings. Empirical results demonstrate that DADAM works also well in practice and compares favorably to competing online optimization methods.Adaptive gradient method. Online learning. Distributed optimization. Regret minimization.

show abstract