A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Li, Junyi; Gu, Bohong; Huang, Heng

doi:10.48550/arxiv.2112.04660

Cited by 4 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…solve the inner problem with one step per hyper-iteration. [19,22,26,25,28,6,63,24,32]. Meanwhile, there are also works utilizing other strategies like penalty methods [42], and also other formulations like the case where the inner problem has non-unique minimizers [31].…”

Section: Related Workmentioning

confidence: 99%

Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction

Li¹,

Huang²,

Huang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms and has been applied to many machine learning tasks such as data cleaning, few-shot learning, and neural architecture search. However, little attention has been paid to solve the bilevel problems under distributed setting. Federated learning (FL) is an emerging paradigm which solves machine learning tasks over distributed-located data. FL problems are challenging to solve due to the heterogeneity and communication bottleneck. However, it is unclear how these challenges will affect the convergence of Bilevel Optimization algorithms. In this paper, we study Federated Bilevel Optimization problems. Specifically, we first propose the FedBiO, a deterministic gradient-based algorithm and we show it requires O(ǫ −2 ) number of iterations to reach an ǫ-stationary point. Then we propose FedBiOAcc to accelerate FedBiO with the momentum-based variance-reduction technique under the stochastic scenario. We show FedBiOAcc has complexity of O(ǫ −1.5 ). Finally, we validate our proposed algorithms via the important Fair Federated Learning task. More specifically, we define a bilevel-based group fair FL objective. Our algorithms show superior performances compared to other baselines in numerical experiments.

show abstract

Section: Related Workmentioning

confidence: 99%

Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction

Li¹,

Huang²,

Huang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…2021b) stepsizes, or by employing larger and complexity dependent mini-batches (Ji et al, 2021). Warm-starting also the LS can further improve the sample-complexity to O(ǫ −2 ) (Arbel and Mairal, 2021;Li et al, 2021). The complexity O(ǫ −2 ) is optimal, since the optimal sample complexity of methods using unbiased stochastic gradient oracles with bounded Table 1: Sample complexity (SC) of stochastic bilevel optimization methods for finding an ǫstationary point of Problem 2 with LL of type 3.…”

Section: Ls Solvermentioning

confidence: 99%

“…A common procedure to improve the overall performance of bilevel algorithms is that of using the LL (or LS) approximate solution found at the (s − 1)-th UL iteration as a starting point for the LL (or LS) solver at the s-th UL iteration. This strategy, which is called warm-start, reduces the number of LL (or LS) iterations needed by the bilevel procedure and is thought to be fundamental to achieve the optimal sample complexity (Arbel and Mairal, 2021;Li et al, 2021). Furthermore, in the stochastic setting (2), warm-start is often accompanied by the use of large mini-batches, i.e.…”

Section: Introductionmentioning

confidence: 99%

Bilevel Optimization with a Lower-level Contraction: Optimal Sample Complexity without Warm-Start

Grazzi¹,

Pontil²,

Salzo³

2022

Preprint

View full text Add to dashboard Cite

We analyze a general class of bilevel problems, in which the upper-level problem consists in the minimization of a smooth objective function and the lower-level problem is to find the fixed point of a smooth contraction map. This type of problems include instances of meta-learning, hyperparameter optimization and data poisoning adversarial attacks. Several recent works have proposed algorithms which warm-start the lower-level problem, i.e. they use the previous lower-level approximate solution as a staring point for the lowerlevel solver. This warm-start procedure allows one to improve the sample complexity in both the stochastic and deterministic settings, achieving in some cases the order-wise optimal sample complexity. We show that without warm-start, it is still possible to achieve order-wise optimal and near-optimal sample complexity for the stochastic and deterministic settings, respectively. In particular, we propose a simple method which uses stochastic fixed point iterations at the lower-level and projected inexact gradient descent at the upper-level, that reaches an ǫ-stationary point using O(ǫ −2 ) and Õ(ǫ −1 ) samples for the stochastic and the deterministic setting, respectively. Compared to methods using warm-start, ours is better suited for meta-learning and yields a simpler analysis that does not need to study the coupled interactions between the upper-level and lower-level iterates.

show abstract

“…Based on different approaches to the estimation of hypergradient, these methods are divided into two categories, i.e. Approximate Implicit Differentiation (AID) [11,14,15,18,22,26,58] and Iterative Differentiation (ITD) [9,10,34,42]. ITD methods first solve the lower level problem approximately and then calculate the hypergradient with backward (forward) automatic differentiation, while AID methods approximate the exact hypergradient [11,19,30,33].…”

Section: Related Workmentioning

confidence: 99%

“…Bilevel optimization problems [44,48,56] involve two levels of problems: an inner problem and an outer problem. Efficient gradient-based alternative update algorithms [15,18,26] have recently been proposed to solve non-distributed bilevel problems, but efficient algorithms designed for the FL setting have not yet been shown. In fact, the most challenging step is to evaluate the hypergradient (gradient w.r.t the variable of the outer problem).…”

Section: Introductionmentioning

confidence: 99%

Communication-Efficient Robust Federated Learning with Noisy Labels

Li,

Pei,

Huang

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) is a promising privacy-preserving machine learning paradigm over distributed located data. In FL, the data is kept locally by each user. This protects the user privacy, but also makes the server difficult to verify data quality, especially if the data are correctly labeled. Training with corrupted labels is harmful to the federated learning task; however, little attention has been paid to FL in the case of label noise. In this paper, we focus on this problem and propose a learning-based reweighting approach to mitigate the effect of noisy labels in FL. More precisely, we tuned a weight for each training sample such that the learned model has optimal generalization performance over a validation set. More formally, the process can be formulated as a Federated Bilevel Optimization problem. Bilevel optimization problem is a type of optimization problem with two levels of entangled problems. The non-distributed bilevel problems have witnessed notable progress recently with new efficient algorithms. However, solving bilevel optimization problems under the Federated Learning setting is under-investigated. We identify that the high communication cost in hypergradient evaluation is the major bottleneck. So we propose Comm-FedBiO to solve the general Federated Bilevel Optimization problems; more specifically, we propose two communication-efficient subroutines to estimate the hypergradient. Convergence analysis of the proposed algorithms is also provided. Finally, we apply the proposed algorithms to solve the noisy label problem. Our approach has shown superior performance on several real-world datasets compared to various baselines. CCS CONCEPTS• Computing methodologies → Supervised learning.

show abstract

A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

Cited by 4 publications

References 22 publications

Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction

Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction

Bilevel Optimization with a Lower-level Contraction: Optimal Sample Complexity without Warm-Start

Communication-Efficient Robust Federated Learning with Noisy Labels

Contact Info

Product

Resources

About