Probabilistic Line Searches for Stochastic Optimization

Abstract

In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a probabilistic line search by combining the structure of existing deterministic methods with notions from Bayesian optimization. Our method retains a Gaussian process surrogate of the univariate optimization objective, and uses a probabilistic belief over the Wolfe conditions to monitor the descent. The algorithm has very low computational cost, and no user-controlled parameters. Experiments show that it effectively removes the need to define a learning rate for stochastic gradient descent.

Barzilai-Borwein Step Size for Stochastic Gradient Descent

“…[22] suggests a line search technique on the component function f i k (x) selected in each iteration, to estimate step size for SAG. [13] suggests performing line search for an estimated function, which is evaluated by a Gaussian process with samples f it (x t ). [14] suggests to generate the step sizes by a given function with an unknown parameter, and to use the online SGD to update this unknown parameter.…”