Simple bilevel programming and extensions

Dempe, Stephan; Dinh, N.; Dutta, Joydeep; Pandit, Tanushree

doi:10.1007/s10107-020-01509-x

Cited by 10 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gong et al [16] also applied their approach to solve a lexicographical optimization of form min θ f (θ) s.t. θ ∈ arg min θ g(θ ), which is a bilevel optimization without an outer variable (known as simple bilevel optimization [8]). Our method is an extension of their method to general bilevel optimization.…”

Section: Related Workmentioning

confidence: 99%

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

Ye¹,

Liu²,

Wright³

et al. 2022

Preprint

View full text Add to dashboard Cite

Bilevel optimization (BO) is useful for solving a variety of important machine learning problems including but not limited to hyperparameter optimization, metalearning, continual learning, and reinforcement learning. Conventional BO methods need to differentiate through the low-level optimization process with implicit differentiation, which requires expensive calculations related to the Hessian matrix. There has been a recent quest for first-order methods for BO, but the methods proposed to date tend to be complicated and impractical for large-scale deep learning applications. In this work, we propose a simple first-order BO algorithm that depends only on first-order gradient information, requires no implicit differentiation, and is practical and efficient for large-scale non-convex functions in deep learning. We provide non-asymptotic convergence analysis of the proposed method to stationary points for non-convex objectives and present empirical results that show its superior practical performance. * Equal contribution. MY mainly contributes on developing theory and BL mainly contributes on conducting experiment. Both authors contribute equally on paper writing.36th Conference on Neural Information Processing Systems (NeurIPS 2022).

show abstract

Section: Related Workmentioning

confidence: 99%

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

Ye¹,

Liu²,

Wright³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It is known that as σ → 0, any cluster point of the solutions of the regularized single-level problem is a solution to the original bilevel problem in (1). Moreover, under certain assumptions [FT08;DDDP21], the solution set of Problem (1) exactly matches with the regularized problem for σ small enough. However, checking such conditions and finding the threshold are often difficult in practice.…”

Section: Introductionmentioning

confidence: 92%

Generalized Frank-Wolfe Algorithm for Bilevel Optimization

Jiang¹,

Abolfazli²,

Mokhtari³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we study a class of bilevel optimization problems, also known as simple bilevel optimization, where we minimize a smooth objective function over the optimal solution set of another convex constrained optimization problem. Several iterative methods have been developed for tackling this class of problems. Alas, their convergence guarantees are not satisfactory as they are either asymptotic for the upper-level objective, or the convergence rates are slow and sub-optimal. To address this issue, in this paper, we introduce a generalization of the Frank-Wolfe (FW) method to solve the considered problem. The main idea of our method is to locally approximate the solution set of the lower-level problem via a cutting plane, and then run a FW-type update to decrease the upper-level objective. When the upper-level objective is convex, we show that our method requires O(max{1/ f , 1/ g }) iterations to find a solution that is foptimal for the upper-level objective and g -optimal for the lower-level objective. Moreover, when the upper-level objective is non-convex, our method requires O(max{1/ 2 f , 1/( f g )}) iterations to find an ( f , g )-optimal solution. We further prove stronger convergence guarantees under the Hölderian error bound assumption on the lower-level problem. To the best of our knowledge, our method achieves the best-known iteration complexity for the considered bilevel problem. We also present numerical experiments to showcase the superior performance of our method compared with state-of-the-art methods.

show abstract

“…Clearly, x(λ) = 0 for all λ > 0. It is also important to note that, possibly unlike the finite termination property (4), the partial calmness concept was introduced as a qualification condition to derive necessary optimality conditions for problem (9); see [15,35] for some papers where this concept is used, and also the papers [14,17] for new results on simple bilevel optimization problems from the perspective of standard bilevel optimization.…”

Section: Standard Bilevel Optimizationmentioning

confidence: 99%

An inertial extrapolation method for convex simple bilevel optimization

Shehu

Vuong

Zemkoho

2019

Optimization Methods and Software

View full text Add to dashboard Cite

We consider a scalar objective minimization problem over the solution set of another optimization problem. This problem is known as a simple bilevel optimization problem and has drawn a significant attention in the last few years. Our inner problem consists of minimizing the sum of smooth and non-smooth functions while the outer one is the minimization of a smooth convex function. We first formulate and give strong convergence analysis of an inertial algorithm for fixedpoint problem of a non-expansive operator in an infinite dimensional Hilbert space. Then we convert the simple bilevel optimization problem to a fixed-point problem of a non-expansive operator in finite dimensional space and design the corresponding algorithm and establish its convergence. Our numerical experiments show that the proposed method in this paper outperforms the currently known best algorithm to solve the class of bilevel optimization problem considered.

show abstract

Simple bilevel programming and extensions

Cited by 10 publications

References 23 publications

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

Generalized Frank-Wolfe Algorithm for Bilevel Optimization

An inertial extrapolation method for convex simple bilevel optimization

Contact Info

Product

Resources

About