SAN: Stochastic Average Newton Algorithm for Minimizing Finite Sums

Chen, Jiabin; Yuan, Rui; Garrigos, Guillaume; Gower, Robert M.

doi:10.48550/arxiv.2106.10520

Cited by 1 publication

(1 citation statement)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Practical experiments have shown that SAG requires many parameter fine-tuning to perform perfectly. Some other variants of SAG are optimization of a finite sum of smooth convex functions (Schmidt et al, 2017) and its second-order version named Stochastic Average Newton (SAN) (Chen et al, 2021).…”

Section: Stochastic Average Gradientmentioning

confidence: 99%

KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey

Ghojogh,

Ghodsi,

Karray

et al. 2021

Preprint

View full text Add to dashboard Cite

This is a tutorial and survey paper on Karush-Kuhn-Tucker (KKT) conditions, first-order and second-order numerical optimization, and distributed optimization. After a brief review of history of optimization, we start with some preliminaries on properties of sets, norms, functions, and concepts of optimization. Then, we introduce the optimization problem, standard optimization problems (including linear programming, quadratic programming, and semidefinite programming), and convex problems. We also introduce some techniques such as eliminating inequality, equality, and set constraints, adding slack variables, and epigraph form. We introduce Lagrangian function, dual variables, KKT conditions (including primal feasibility, dual feasibility, weak and strong duality, complementary slackness, and stationarity condition), and solving optimization by method of Lagrange multipliers. Then, we cover first-order optimization including gradient descent, line-search, convergence of gradient methods, momentum, steepest descent, and backpropagation. Other first-order methods are explained, such as accelerated gradient method, stochastic gradient descent, mini-batch gradient descent, stochastic average gradient, stochastic variance reduced gradient, Ada-Grad, RMSProp, and Adam optimizer, proximal methods (including proximal mapping, proximal point algorithm, and proximal gradient method), and constrained gradient methods (including projected gradient method, projection onto convex sets, and Frank-Wolfe method). We also cover non-smooth and 1 optimization methods including lasso regularization, convex conjugate, Huber function, soft-thresholding, coordinate descent, and subgradient methods. Then, we explain second-order methods including Newton's method for unconstrained, equality constrained, and inequality constrained problems. We explain the interior-point method, barrier methods, Wolfe conditions for line-search, fast solving system of equations (including decomposition methods and conjugate gradient), and quasi-Newton's method (including BFGS, LBFGS, Broyden, DFP, and SR1 methods). The sequential convex programming for non-convex optimization is also introduced. Finally, we explain distributed optimization including alternating optimization, dual decomposition methods, augmented Lagrangian, and alternating direction method of multipliers (ADMM). We also introduce some techniques for using ADMM for many constraints and variables.

show abstract

Section: Stochastic Average Gradientmentioning

confidence: 99%