Statistical Inference for the Population Landscape via Moment-Adjusted Stochastic Gradients

Liang, Tengyuan; Su, Weijie

doi:10.1111/rssb.12313

Cited by 8 publications

(3 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, Fang et al [2018], Fang [2019] proposed online bootstrap procedures for the estimation of confidence intervals via randomly perturbed SGD. Meanwhile, , Su and Zhu [2018], Liang and Su [2019] proposed variants of the SGD algorithm to facilitate inference in a non-asymptotic fashion.…”

Section: Related Workmentioning

confidence: 99%

Statistical Estimation and Inference via Local SGD in Federated Learning

Li¹,

Liang²,

Chang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) makes a large amount of edge computing devices (e.g., mobile phones) jointly learn a global model without data sharing. In FL, data are generated in a decentralized manner with high heterogeneity. This paper studies how to perform statistical estimation and inference in the federated setting. We analyze the so-called Local SGD, a multi-round estimation procedure that uses intermittent communication to improve communication efficiency. We first establish a functional central limit theorem that shows the averaged iterates of Local SGD weakly converge to a rescaled Brownian motion. We next provide two iterative inference methods: the plug-in and the random scaling. Random scaling constructs an asymptotically pivotal statistic for inference by using the information along the whole Local SGD path. Both the methods are communication efficient and applicable to online data. Our theoretical and empirical results show that Local SGD simultaneously achieves both statistical efficiency and communication efficiency.

show abstract

Section: Related Workmentioning

confidence: 99%

Statistical Estimation and Inference via Local SGD in Federated Learning

Li¹,

Liang²,

Chang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Kingma and Ba (2015) introduced Adam, computing adaptive learning rates for different parameters from estimates of first and second moments of the gradients. Liang and Su (2019) employed a similar idea as AdaGrad to adjust the gradient direction and showed that the distribution for inference can be simulated iteratively. Toulis and Airoldi (2017) developed implicit SGD procedures and established the resulting estimator's asymptotic normality.…”

Section: Introductionmentioning

confidence: 99%

Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

Lee

Liao

Seo

et al. 2022

AAAI

View full text Add to dashboard Cite

We develop a new method of online inference for a vector of parameters estimated by the Polyak-Ruppert averaging procedure of stochastic gradient descent (SGD) algorithms. We leverage insights from time series regression in econometrics and construct asymptotically pivotal statistics via random scaling. Our approach is fully operational with online data and is rigorously underpinned by a functional central limit theorem. Our proposed inference method has a couple of key advantages over the existing methods. First, the test statistic is computed in an online fashion with only SGD iterates and the critical values can be obtained without any resampling methods, thereby allowing for efficient implementation suitable for massive online data. Second, there is no need to estimate the asymptotic variance and our inference method is shown to be robust to changes in the tuning parameters for SGD algorithms in simulation experiments with synthetic data.

show abstract

“…Fang et al (2018) and Fang (2019) proposed bootstrap procedures for constructing confidence intervals through perturbed-SGD. Meanwhile, variants of SGD algorithm and corresponding inference in non-asymptotic fashion are studied in Su and Zhu (2018) and Liang and Su (2019).…”

Section: Introductionmentioning

confidence: 99%

Online Covariance Matrix Estimation in Stochastic Gradient Descent

Zhu¹,

Chen²,

Wu³

2020

Preprint

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) algorithm is widely used for parameter estimation especially in online setting. While this recursive algorithm is popular for computation and memory efficiency, the problem of quantifying variability and randomness of the solutions has been rarely studied. This paper aims at conducting statistical inference of SGD-based estimates in online setting. In particular, we propose a fully online estimator for the covariance matrix of averaged SGD iterates (ASGD). Based on the classic asymptotic normality results of ASGD, we construct asymptotically valid confidence intervals for model parameters. Upon receiving new observations, we can quickly update the covariance estimator and confidence intervals. This approach fits in online setting even if the total number of data is unknown and takes the full advantage of SGD: efficiency in both computation and memory.

show abstract

Statistical Inference for the Population Landscape via Moment-Adjusted Stochastic Gradients

Cited by 8 publications

References 49 publications

Statistical Estimation and Inference via Local SGD in Federated Learning

Statistical Estimation and Inference via Local SGD in Federated Learning

Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

Online Covariance Matrix Estimation in Stochastic Gradient Descent

Contact Info

Product

Resources

About