Bilevel stochastic methods for optimization and machine learning: Bilevel stochastic descent and DARTS

Giovannelli, Tommaso; Kent, Griffin; Vicente, Luís Nunes

doi:10.48550/arxiv.2110.00604

Cited by 3 publications

(9 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Baselines A comprehensive set of state-of-the-art BO methods are chosen as baseline methods. This includes the fully first-order methods: BSG-1 [15] and BVFSM [34], ; a stationary-seeking method: Penalty [39], explicit/implicit methods: ITD [22], AID-CG (using conjugate gradient), AID-FP (using fixed point method) [17], reverse (using reverse auto-differentiation) [11] stocBiO [22], and VRBO [51].…”

Section: Methodsmentioning

confidence: 99%

“…Figure 3: Trajectories of (v k , θ k ) on the toy coreset problem (6.1) obtained from BOME (blue) and three recent first-order bilevel methods: BSG-1 [15] (green), BVFSM [34] (orange), and Penalty [39] (red). The goal of the problem is to find the closet point (marked by opt.)…”

Section: A1 Toy Coreset Problemmentioning

confidence: 99%

“…where the last inequality is by the assumption on ξ ≤ 1/L. To show the second inequality, we use the complementary slackness of Problem (15), that is…”

Section: C4 Proof Of Lemma 12mentioning

confidence: 99%

“…This still requires Hessian information, and more importantly, is unsuitable for nonconvex g since it allows θ to be any stationary point of g(v, •), not necessarily a minimizer. To the best of our knowledge, the only existing fully first-order BO algorithms 2 are BSG-1 [15] and BVFSM with its variants [33][34][35]; but BSG-1 relies on a nonvanishing approximation that does not yield convergence to the correct solution in general, and BVFSM is sensitive to hyper-parameters on large-scale practical problems and lacks a complete non-asymptotic analysis for the practically implemented algorithm.…”

Section: Introductionmentioning

confidence: 99%

“…Other examples of approximation methods include a neural surrogate method which approximates θ * (v) and its gradient ∇ v θ * (v) with neural networks [38] and Newton-Gaussian approximation of the Hessian matrix with covariance of gradient [15]. Both approaches introduce non-vanishing approximation error that is difficult to control.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

Ye¹,

Liu²,

Wright³

et al. 2022

Preprint

View full text Add to dashboard Cite

Bilevel optimization (BO) is useful for solving a variety of important machine learning problems including but not limited to hyperparameter optimization, metalearning, continual learning, and reinforcement learning. Conventional BO methods need to differentiate through the low-level optimization process with implicit differentiation, which requires expensive calculations related to the Hessian matrix. There has been a recent quest for first-order methods for BO, but the methods proposed to date tend to be complicated and impractical for large-scale deep learning applications. In this work, we propose a simple first-order BO algorithm that depends only on first-order gradient information, requires no implicit differentiation, and is practical and efficient for large-scale non-convex functions in deep learning. We provide non-asymptotic convergence analysis of the proposed method to stationary points for non-convex objectives and present empirical results that show its superior practical performance. * Equal contribution. MY mainly contributes on developing theory and BL mainly contributes on conducting experiment. Both authors contribute equally on paper writing.36th Conference on Neural Information Processing Systems (NeurIPS 2022).

show abstract

Section: Methodsmentioning

confidence: 99%

Section: A1 Toy Coreset Problemmentioning

confidence: 99%

“…where the last inequality is by the assumption on ξ ≤ 1/L. To show the second inequality, we use the complementary slackness of Problem (15), that is…”

Section: C4 Proof Of Lemma 12mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

Ye¹,

Liu²,

Wright³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Algorithms for Simple Bilevel Programming

Dutta

Pandit

2020

Springer Optimization and Its Applications

View full text Add to dashboard Cite

We develop new algorithms for Riemannian bilevel optimization. We focus in particular on batch and stochastic gradient-based methods, with the explicit goal of avoiding second-order information such as Riemannian hyper-gradients. We propose and analyze RF 2 SA, a method that leverages first-order gradient information to navigate the complex geometry of Riemannian manifolds efficiently. Notably, RF 2 SA is a singleloop algorithm, and thus easier to implement and use. Under various setups, including stochastic optimization, we provide explicit convergence rates for reaching ϵ-stationary points. We also address the challenge of optimizing over Riemannian manifolds with constraints by adjusting the multiplier in the Lagrangian, ensuring convergence to the desired solution without requiring access to second-order derivatives.

show abstract

A Fully First-Order Method for Stochastic Bilevel Optimization

Kwon¹,

Kwon²,

Wright³

et al. 2023

Preprint

View full text Add to dashboard Cite

We consider stochastic unconstrained bilevel optimization problems when only the first-order gradient oracles are available. While numerous optimization methods have been proposed for tackling bilevel problems, existing methods either tend to require possibly expensive calculations involving Hessians of lower-level objectives, or lack rigorous finite-time performance guarantees. In this work, we propose a Fully First-order Stochastic Approximation (F 2 SA) method, and study its non-asymptotic convergence properties. Specifically, we show that F 2 SA converges to an -stationary solution of the bilevel problem after −7/2 , −5/2 , or −3/2 iterations (each iteration using O(1) samples) when stochastic noises are in both level objectives, only in the upper-level objective, or not present (deterministic settings), respectively. We further show that if we employ momentum-assisted gradient estimators, the iteration complexities can be improved to −5/2 , −4/2 , or −3/2 , respectively. We demonstrate the superior practical performance of the proposed method over existing second-order based approaches on MNIST data-hypercleaning experiments.

show abstract

Bilevel stochastic methods for optimization and machine learning: Bilevel stochastic descent and DARTS

Cited by 3 publications

References 29 publications

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach

Algorithms for Simple Bilevel Programming

A Fully First-Order Method for Stochastic Bilevel Optimization

Contact Info

Product

Resources

About