Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming

Dai, Yu‐Hong; Fletcher, R.

doi:10.1007/s00211-004-0569-y

Cited by 322 publications

(257 citation statements)

References 32 publications

Supporting

Mentioning

253

Contrasting

Unclassified

Order By: Relevance

“…One could employ iterative shrinkage [12] or the gradient projection method [24,5] for the prediction phase; in this paper we chose a special form of the latter. For the subspace phase, one could minimize F S [17,9] instead of a quadratic model of this function [8,20], but we choose to work with a model due to the high cost of evaluating the objective function.…”

Section: A Newton-cg Methods For L 1 Regularized Modelsmentioning

confidence: 99%

Sample size selection in optimization methods for machine learning

et al. 2012

View full text Add to dashboard Cite

This paper presents a methodology for using varying sample sizes in batch-type optimization methods for large scale machine learning problems. The first part of the paper deals with the delicate issue of dynamic sample selection in the evaluation of the function and gradient. We propose a criterion for increasing the sample size based on variance estimates obtained during the computation of a batch gradient. We establish an O(1/ ) complexity bound on the total cost of a gradient method. The second part of the paper describes a practical Newton method that uses a smaller sample to compute Hessian vector-products than to evaluate the function and the gradient, and that also employs a dynamic sampling technique. The focus of the paper shifts in the third part of the paper to L 1 regularized problems designed to produce sparse solutions. We propose a Newton-like method that consists of two phases: a (minimalistic) gradient projection phase that identifies zero variables, and subspace phase that applies a subsampled Hessian Newton iteration in the free variables. Numerical tests on speech recognition problems illustrate the performance of the algorithms.

show abstract

Section: A Newton-cg Methods For L 1 Regularized Modelsmentioning

confidence: 99%

Sample size selection in optimization methods for machine learning

et al. 2012

View full text Add to dashboard Cite

show abstract

“…Since CBB is invariant under an orthogonal transformation and since gradient components corresponding to identical eigenvalues can be combined (see, e.g. Dai & Fletcher, 2005b), we assume without loss of generality that A is diagonal:…”

Section: The Cbb Methods For Convex Quadratic Programmingmentioning

confidence: 99%

“…An even more adaptive way of choosing f r is proposed by Toint (1997) for trust region algorithms and then extended by Dai & Zhang (2001). Compared with (4.5), the new adaptive way of choosing f r allows big jumps in function values, and is therefore very suitable for the BB algorithm (see Dai & Fletcher, 2005bDai & Zhang, 2001).…”

Section: Non-monotone Line Search and Cycle Numbermentioning

confidence: 99%

The cyclic Barzilai-–Borwein method for unconstrained optimization

Dai¹,

Hager²,

Schittkowski³

et al. 2006

IMA Journal of Numerical Analysis

Self Cite

175

163

View full text Add to dashboard Cite

In the cyclic Barzilai-Borwein (CBB) method, the same Barzilai-Borwein (BB) stepsize is reused for m consecutive iterations. It is proved that CBB is locally linearly convergent at a local minimizer with positive definite Hessian. Numerical evidence indicates that when m > n/2 3, where n is the problem dimension, CBB is locally superlinearly convergent. In the special case m = 3 and n = 2, it is proved that the convergence rate is no better than linear, in general. An implementation of the CBB method, called adaptive cyclic Barzilai-Borwein (ACBB), combines a non-monotone line search and an adaptive choice for the cycle length m. In numerical experiments using the CUTEr test problem library, ACBB performs better than the existing BB gradient algorithm, while it is competitive with the well-known PRP+ conjugate gradient algorithm.

show abstract

“…Numerous variants have been proposed recently, and subjected to with theoretical and computational analysis. The BB step-length rules have also been extended to constrained optimization, particularly to bound-constrained quadratic programming; see, for example [10] and [18]. The same formulae (24) and (25) can be used in these cases to determine the step length, but we obtain x k+1 by projecting x k − α k ∇F (x k ) onto the feasible set X, and possibly performing additional backtracking or line-search modifications to ensure descent in F .…”

Section: Barzilai-borwein Strategiesmentioning

confidence: 99%

Duality-based algorithms for total-variation-regularized image restoration

Zhu¹,

Wright

Chan³

2008

Comput Optim Appl

159

122

View full text Add to dashboard Cite

Image restoration models based on total variation (TV) have become popular since their introduction by Rudin, Osher, and Fatemi (ROF) in 1992. The dual formulation of this model has a quadratic objective with separable constraints, making projections onto the feasible set easy to compute. This paper proposes application of gradient projection (GP) algorithms to the dual formulation. We test variants of GP with different step length selection and line search strategies, including techniques based on the Barzilai-Borwein method. Global convergence can in some cases be proved by appealing to existing theory. We also propose a sequential quadratic programming (SQP) approach that takes account of the curvature of the boundary of the dual feasible set. Computational experiments show that the proposed approaches perform well in a wide range of applications and that some are significantly faster than previously proposed methods, particularly when only modest accuracy in the solution is required.

show abstract

Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming

Cited by 322 publications

References 32 publications

Sample size selection in optimization methods for machine learning

Sample size selection in optimization methods for machine learning

The cyclic Barzilai-–Borwein method for unconstrained optimization

Duality-based algorithms for total-variation-regularized image restoration

Contact Info

Product

Resources

About