Spectral Properties of Barzilai--Borwein Rules in Solving Singly Linearly Constrained Optimization Problems Subject to Lower and Upper Bounds

Abstract

In 1988, Barzilai and Borwein published a pioneering paper which opened the way to inexpensively accelerate first-order. In more detail, in the framework of unconstrained optimization, Barzilai and Borwein developed two strategies to select the step length in gradient descent methods with the aim of encoding some second-order information of the problem without computing and/or employing the Hessian matrix of the objective function. Starting from these ideas, several efficient step length techniques have been suggested in the last decades in order to make gradient descent methods more and also more appealing for problems which handle large-scale data and require realtime solutions. Typically, these new step length selection rules have been tuned in the quadratic unconstrained framework for sweeping the spectrum of the Hessian matrix, and then applied also to nonquadratic constrained problems, without any substantial modification, by showing them to be very effective anyway. In this paper, we deeply analyze how, in quadratic and nonquadratic minimization problems, the presence of a feasible region, expressed by a single linear equality constraint together with lower and upper bounds, influences the spectral properties of the original Barzilai-Borwein (BB) rules, generalizing recent results provided for box-constrained quadratic problems. This analysis gives rise to modified BB approaches able not only to capture second-order information but also to exploit the nature of the feasible region. We show the benefits gained by the new step length rules on a set of test problems arising also from machine learning and image processing applications.

Equipping Barzilai-Borwein method with two dimensional quadratic termination property

“…Numerical experiments on minimizing quadratic functions (4) show that our method performs much better than the most successful gradient methods developed in recent literature including BB1 [1], Dai-Yuan (DY) [14], ABB [46], ABBmin1 [20], ABBmin2 [20] and SDC [16]. In addition, numerical comparisons of our method with the spectral projected gradient (SPG) method [2,3] on box-constrained problems from the CUTEst collection [24] and with the Dai-Fletcher [11] and EQ-VABBmin methods [7] on random SLB problems and SLB problems arising in support vector machine (SVM) [6] highly suggest the potential benefits of our new proposed method for solving general box-constrained and SLB optimization problems.…”

“…Both the above two schemes adopt short and long stepsizes, and perform better than using only one type of stepsize. Variants and other stepsize rules can be found in [4,7,9,20,21,29,37]. It has been observed that replacing α BB2 k with a short stepsize will further improve the performance of the adaptive scheme [8,20,29].…”

“…where m is a positive integer. As for the adaptive scheme, the authors of [4,7] suggested to update τ by…”

“…for some γ ≥ 1. Such a dynamic strategy has the advantage of less dependent on the value of τ while keeping the efficiency of the stepsize [4,7]. Based on the rule (31), we suggest the following adaptive scheme…”

Spatially Adaptive Regularization in Image Segmentation

“…Our techniques are based on the so-called cartoon-texture decomposition of the given image, on the mean and median filters, and on a thresholding technique. The resulting locally adaptive segmentation model can be solved either by smoothing the discrete TV term-see, e.g., [16,17]-and applying optimization solvers for differentiable problems, such as spectral gradient methods [18][19][20][21], or by using directly optimization solvers for nondifferentiable problems, such as Bregman, proximal and ADMM methods [22][23][24][25][26][27][28]. In this work, we use an alternating minimization procedure exploiting the split Bregman (SB) method proposed in [24].…”