Statistical Estimation and Inference via Local SGD in Federated Learning

Li, Xiang; Liang, Jiadong; Chang, Xiangyu; Zhang, Zhihua

doi:10.48550/arxiv.2109.01326

Cited by 2 publications

(3 citation statements)

References 28 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Yuan & Ma (2020) proposed an accelerated FedAvg algorithm, which requires O(K 1/3 poly log(T )) to attain linear speedup. Recently, Li et al (2021) investigated the statistical estimation and inference problem for local SGD in FL. However, Li et al (2021) focused on the unconstrained smooth statistical optimization, but we considered a different problem with non-smooth regularizer aiming to recover the sparse/low-rank structure of groundtruth model.…”

Section: A Related Workmentioning

confidence: 99%

“…Recently, Li et al (2021) investigated the statistical estimation and inference problem for local SGD in FL. However, Li et al (2021) focused on the unconstrained smooth statistical optimization, but we considered a different problem with non-smooth regularizer aiming to recover the sparse/low-rank structure of groundtruth model. For the strongly convex finite-sum problem, Mitra et al (2021) proposed an algorithm named FedLin based on the variance reduction technique and obtained the linear convergence rate.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Fast Composite Optimization and Statistical Recovery in Federated Learning

Bao¹,

Crawshaw²,

Si³

et al. 2022

Preprint

View full text Add to dashboard Cite

As a prevalent distributed learning paradigm, Federated Learning (FL) trains a global model on a massive amount of devices with infrequent communication. This paper investigates a class of composite optimization and statistical recovery problems in the FL setting, whose loss function consists of a data-dependent smooth loss and a non-smooth regularizer. Examples include sparse linear regression using Lasso, low-rank matrix recovery using nuclear norm regularization, etc. In the existing literature, federated composite optimization algorithms are designed only from an optimization perspective without any statistical guarantees. In addition, they do not consider commonly used (restricted) strong convexity in statistical recovery problems. We advance the frontiers of this problem from both optimization and statistical perspectives. From optimization upfront, we propose a new algorithm named Fast Federated Dual Averaging for strongly convex and smooth loss and establish state-of-the-art iteration and communication complexity in the composite setting. In particular, we prove that it enjoys a fast rate, linear speedup, and reduced communication rounds. From statistical upfront, for restricted strongly convex and smooth loss, we design another algorithm, namely Multi-stage Federated Dual Averaging, and prove a high probability complexity bound with linear speedup up to optimal statistical precision. Experiments in both synthetic and real data demonstrate that our methods perform better than other baselines. To the best of our This is a revised version to fix the imprecise statements about linear speedup from the ICML proceedings. We use another averaging scheme for the returned solutions in Theorem 2.1 and 3.1 to guarantee linear speedup when the number of iterations is large.

show abstract

Section: A Related Workmentioning

confidence: 99%

Section: A Related Workmentioning

confidence: 99%

Fast Composite Optimization and Statistical Recovery in Federated Learning

Bao¹,

Crawshaw²,

Si³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Table 1 provides a summary comparison between our proposed method and the existing methods. After the first version of our paper appeared on arXiv, Li et al (2021) and Chen et al (2021) applied the idea of random scaling to their SGD inference problems: federated learning for the former and Kiefer-Wofowitz methods for the latter.…”

Section: Introductionmentioning

confidence: 99%

Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

Lee

Liao

Seo

et al. 2022

AAAI

View full text Add to dashboard Cite

We develop a new method of online inference for a vector of parameters estimated by the Polyak-Ruppert averaging procedure of stochastic gradient descent (SGD) algorithms. We leverage insights from time series regression in econometrics and construct asymptotically pivotal statistics via random scaling. Our approach is fully operational with online data and is rigorously underpinned by a functional central limit theorem. Our proposed inference method has a couple of key advantages over the existing methods. First, the test statistic is computed in an online fashion with only SGD iterates and the critical values can be obtained without any resampling methods, thereby allowing for efficient implementation suitable for massive online data. Second, there is no need to estimate the asymptotic variance and our inference method is shown to be robust to changes in the tuning parameters for SGD algorithms in simulation experiments with synthetic data.

show abstract

Statistical Estimation and Inference via Local SGD in Federated Learning

Cited by 2 publications

References 28 publications

Fast Composite Optimization and Statistical Recovery in Federated Learning

Fast Composite Optimization and Statistical Recovery in Federated Learning

Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

Contact Info

Product

Resources

About