Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction

Li, Junyi; Huang, Feihu; Huang, Heng

doi:10.48550/arxiv.2205.01608

Cited by 4 publications

(11 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We are ready to prove the main results in Theorem A. 16. We first summarize main results in Lemmas A.19, A.8 and A.7:…”

Section: A11 Case 1: Assumption 24 Holdsmentioning

confidence: 94%

“…In this section we will prove the following convergence result of the DBO algorithm: Theorem A. 16. In Algorithm 3, suppose Assumptions 2.1, 2.2, and 2.3 hold.…”

Section: A1 Proof Of the Dbo Convergencementioning

confidence: 99%

“…There exist recent works considering bilevel optimization under distributed setting. Bilevel optimization under a federated setting has received some attention recently [35,16], and so does min-max optimization under various distributed settings [36,19,31]. However, none of these papers considers bilevel optimization under the decentralized setting.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Decentralized Bilevel Optimization

Chen¹,

Huang²,

Ma³

2022

Preprint

View full text Add to dashboard Cite

Bilevel optimization has been successfully applied to many important machine learning problems. Algorithms for solving bilevel optimization have been studied under various settings. In this paper, we study the nonconvex-strongly-convex bilevel optimization under a decentralized setting. We design decentralized algorithms for both deterministic and stochastic bilevel optimization problems. Moreover, we analyze the convergence rates of the proposed algorithms in difference scenarios including the case where data heterogeneity is observed across agents. Numerical experiments on both synthetic and real data demonstrate that the proposed methods are efficient.

show abstract

“…We are ready to prove the main results in Theorem A. 16. We first summarize main results in Lemmas A.19, A.8 and A.7:…”

Section: A11 Case 1: Assumption 24 Holdsmentioning

confidence: 94%

“…In this section we will prove the following convergence result of the DBO algorithm: Theorem A. 16. In Algorithm 3, suppose Assumptions 2.1, 2.2, and 2.3 hold.…”

Section: A1 Proof Of the Dbo Convergencementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Decentralized Bilevel Optimization

Chen¹,

Huang²,

Ma³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…the inner problem has multiple minimizers [25,49]. A recent work [27] applied momentum-based acceleration to solve federated bilevel optimization problems.…”

Section: Related Workmentioning

confidence: 99%

Communication-Efficient Robust Federated Learning with Noisy Labels

Li,

Pei,

Huang

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) is a promising privacy-preserving machine learning paradigm over distributed located data. In FL, the data is kept locally by each user. This protects the user privacy, but also makes the server difficult to verify data quality, especially if the data are correctly labeled. Training with corrupted labels is harmful to the federated learning task; however, little attention has been paid to FL in the case of label noise. In this paper, we focus on this problem and propose a learning-based reweighting approach to mitigate the effect of noisy labels in FL. More precisely, we tuned a weight for each training sample such that the learned model has optimal generalization performance over a validation set. More formally, the process can be formulated as a Federated Bilevel Optimization problem. Bilevel optimization problem is a type of optimization problem with two levels of entangled problems. The non-distributed bilevel problems have witnessed notable progress recently with new efficient algorithms. However, solving bilevel optimization problems under the Federated Learning setting is under-investigated. We identify that the high communication cost in hypergradient evaluation is the major bottleneck. So we propose Comm-FedBiO to solve the general Federated Bilevel Optimization problems; more specifically, we propose two communication-efficient subroutines to estimate the hypergradient. Convergence analysis of the proposed algorithms is also provided. Finally, we apply the proposed algorithms to solve the noisy label problem. Our approach has shown superior performance on several real-world datasets compared to various baselines. CCS CONCEPTS• Computing methodologies → Supervised learning.

show abstract

“…There are much fewer studies on BO in a distributed setting. The recent works [25,34] consider a BO where both the outer and inner problems are defined with the expectations over data distributed across nodes. They analyze the communication complexity of their methods in a non-convex setting.…”

Section: Related Workmentioning

confidence: 99%

Federated Learning on Adaptively Weighted Nodes by Bilevel Optimization

Huang¹,

Lin²,

Street³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a federated learning method with weighted nodes in which the weights can be modified to optimize the model's performance on a separate validation set. The problem is formulated as a bilevel optimization where the inner problem is a federated learning problem with weighted nodes and the outer problem focuses on optimizing the weights based on the validation performance of the model returned from the inner problem. A communication-efficient federated optimization algorithm is designed to solve this bilevel optimization problem. Under an error-bound assumption, we analyze the generalization performance of the output model and identify scenarios when our method is in theory superior to training a model only locally and to federated learning with static and evenly distributed weights.

show abstract

Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction

Cited by 4 publications

References 49 publications

Decentralized Bilevel Optimization

Decentralized Bilevel Optimization

Communication-Efficient Robust Federated Learning with Noisy Labels

Federated Learning on Adaptively Weighted Nodes by Bilevel Optimization

Contact Info

Product

Resources

About