On the efficiency of Stochastic Quasi-Newton Methods for Deep Learning

Abstract

While first-order methods are popular for solving optimization problems that arise in largescale deep learning problems, they come with some acute deficiencies. To diminish such shortcomings, there has been recent interest in applying second-order methods such as quasi-Newton based methods which construct Hessians approximations using only gradient information. The main focus of our work is to study the behaviour of stochastic quasi-Newton algorithms for training deep neural networks. We have analyzed the performance of two well-known quasi-Newton updates, the limited memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) and the Symmetric Rank One (SR1). This study fills a gap concerning the real performance of both updates and analyzes whether more efficient training is obtained when using the more robust BFGS update or the cheaper SR1 formula which allows for indefinite Hessian approximations and thus can potentially help to better navigate the pathological saddle points present in the non-convex loss functions found in deep learning. We present and discuss the results of an extensive experimental study which includes the effect of batch normalization and network's architecture, the limited memory parameter, the batch size and the type of sampling strategy. we show that stochastic quasi-Newton optimizers are efficient and able to outperform in some instances the well-known first-order Adam optimizer run with the optimal combination of its numerous hyperparameters.

Newton's Method in Matlab 19.0 to Minimize a Function of Direct Solar Infrared Radiation (2100nm to 4000nm)

“…Therefore, Newton's method is used to solve equations with non-differentiable function with sufficient conditions of convergence and estimates of both the speed and the range of convergence [4]. Also there was interest in applying second-order methods, such as Newton-based methods that build Hessian approximations using only gradient in-formation to study the behavior of Newtonian stochastic algorithms to train deep neural networks [5].…”