An investigation of Newton-Sketch and subsampled Newton methods

Berahas, Albert S.; Bollapragada, Raghu; Nocedal, Jorge

doi:10.1080/10556788.2020.1725751

Cited by 65 publications

(59 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When the overlap set O k is not too small, y k is a useful approximation of the curvature of the objective function along the most recent displacement, and leads to a productive quasi-Newton step. This observation is based on an important property of Newton-like methods, namely that there is much more freedom in choosing a Hessian approximation than in computing the gradient [3,8,16,45]. More specifically, a smaller sample O k can be employed for updating the inverse Hessian approximation H k , than for computing the batch gradient g Sk k used to define the search direction −H k g Sk k .…”

Section: A Multi-batch Quasi-newton Methodsmentioning

confidence: 99%

A robust multi-batch L-BFGS method for machine learning

Berahas

Takáč

2019

Optimization Methods and Software

Self Cite

View full text Add to dashboard Cite

This paper describes an implementation of the L-BFGS method designed to deal with two adversarial situations. The first occurs in distributed computing environments where some of the computational nodes devoted to the evaluation of the function and gradient are unable to return results on time. A similar challenge occurs in a multi-batch approach in which the data points used to compute function and gradients are purposely changed at each iteration to accelerate the learning process. Difficulties arise because L-BFGS employs gradient differences to update the Hessian approximations, and when these gradients are computed using different data points the updating process can be unstable. This paper shows how to perform stable quasi-Newton updating in the multi-batch setting, studies the convergence properties for both convex and nonconvex functions, and illustrates the behavior of the algorithm in a distributed computing platform on binary classification logistic regression and neural network training problems that arise in machine learning.

show abstract

Section: A Multi-batch Quasi-newton Methodsmentioning

confidence: 99%

A robust multi-batch L-BFGS method for machine learning

Berahas

Takáč

2019

Optimization Methods and Software

Self Cite

View full text Add to dashboard Cite

show abstract

“…Finally, we would like to mention that in [5] the authors perform a local complexity analysis of subsampled Inexact Newton methods and also show that methods that in-corporate stochastic second-order information can be far more efficient on badly-scaled or ill-conditioned problems than first-order methods.…”

Section: Related Workmentioning

confidence: 99%

“…Our analysis is strictly related to that developed in [5,7,36,37], where convergence of Inexact subsampled Newton methods is investigated both in probability [36,37] and expectation [9,7,5]. We differ from these papers as we focus on the choice of the forcing terms, on the nonmonotone line search strategy and on adaptive choices of Hessian sample size.…”

Section: Related Workmentioning

confidence: 99%

“…One successful example is the stochastic gradient method, that employs a smaller subset of gradient components and thus reduce the cost even further, [19]. On the other hand several papers investigate the use of second order methods in this framework and demonstrate advantages in some important problems if the second order methods are correctly implemented, [4,7,5,10,11,18,36,37,34,40,41]. For a comprehensive discussion of this issue one can see [8] and references cited therein.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Subsampled inexact Newton methods for minimizing large sums of convex functions

Bellavia

Krejić

Jerinkic

2019

IMA Journal of Numerical Analysis

View full text Add to dashboard Cite

This paper deals with the minimization of large sum of convex functions by Inexact Newton (IN) methods employing subsampled functions, gradients and Hessian approximations. The Conjugate Gradient method is used to compute the inexact Newton step and global convergence is enforced by a nonmonotone line search procedure. The aim is to obtain methods with affordable costs and fast convergence. Assuming strongly convex functions, R-linear convergence and worst-case iteration complexity of the procedure are investigated when functions and gradients are approximated with increasing accuracy. A set of rules for the forcing parameters and subsample Hessian sizes are derived that ensure local q-linear/superlinear convergence of the proposed method. The random choice of the Hessian subsample is also considered and convergence in the mean square, both for finite and infinite sums of functions, is proved. Finally, global convergence with asymptotic R-linear rate of IN methods is extended to the case of sum of convex function and strongly convex objective function. Numerical results on well known binary classification problems are also given. Adaptive strategies for selecting forcing terms and Hessian subsample size, streaming out of the theoretical analysis, are employed and the numerical results showed that they yield effective IN methods.

show abstract

“…A fruitful line of research has focused on how to improve the asymptotic convergence rate as t → ∞ through preconditioning: a technique that involves approximating the unknown Hessian H.θ/ = ∇ 2 θ L.θ/ (see, for instance, Bordes et al (2009) and references therein). Utilizing the curvature information that is reflected by various efficient approximations of the Hessian matrix, stochastic quasi-Newton methods (Moritz et al, 2016;Byrd et al, 2016;Wang et al, 2017;Schraudolph et al, 2007;Mokhtari and Ribeiro, 2015;Becker and Fadili, 2012), Newton sketching or subsampled Newton methods (Pilanci and Wainwright, 2015;Xu et al, 2016;Berahas et al, 2017;Bollapragada et al, 2016) and stochastic approximation of the inverse Hessian via Taylor series expansion (Agarwal et al, 2017) have been proposed to strike a balance between convergence rate and per-iteration complexity.…”

Section: Relationships To the Literaturementioning

confidence: 99%

Statistical Inference for the Population Landscape via Moment-Adjusted Stochastic Gradients

Liang

2019

Journal of the Royal Statistical Society Series B: Statistical Methodology

View full text Add to dashboard Cite

Summary Modern statistical inference tasks often require iterative optimization methods to compute the solution. Convergence analysis from an optimization viewpoint informs us only how well the solution is approximated numerically but overlooks the sampling nature of the data. In contrast, recognizing the randomness in the data, statisticians are keen to provide uncertainty quantification, or confidence, for the solution obtained by using iterative optimization methods. The paper makes progress along this direction by introducing moment‐adjusted stochastic gradient descent: a new stochastic optimization method for statistical inference. We establish non‐asymptotic theory that characterizes the statistical distribution for certain iterative methods with optimization guarantees. On the statistical front, the theory allows for model misspecification, with very mild conditions on the data. For optimization, the theory is flexible for both convex and non‐convex cases. Remarkably, the moment adjusting idea motivated from ‘error standardization’ in statistics achieves a similar effect to acceleration in first‐order optimization methods that are used to fit generalized linear models. We also demonstrate this acceleration effect in the non‐convex setting through numerical experiments.

show abstract

An investigation of Newton-Sketch and subsampled Newton methods

Cited by 65 publications

References 12 publications

A robust multi-batch L-BFGS method for machine learning

A robust multi-batch L-BFGS method for machine learning

Subsampled inexact Newton methods for minimizing large sums of convex functions

Statistical Inference for the Population Landscape via Moment-Adjusted Stochastic Gradients

Contact Info

Product

Resources

About