Online Covariance Matrix Estimation in Stochastic Gradient Descent

Zhu, Wanrong; Chen, Xi; Wu, Wei Biao

doi:10.48550/arxiv.2002.03979

Cited by 2 publications

(11 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We emphasize that this is a non-trivial problem even in the first-order setting, as the iterates x t form an inhomogeneous Markov chain. In this section, we leverage the recent work by [ZCW20], who proposed an online estimator of the asymptotic covariance matrix in the stochastic first-order setting, and extend it to the stochastic zeroth-order setting and propose an estimator of the covariance matrix Ṽ appearing in Proposition 2.1.…”

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

“…As suggested in [ZCW20], this estimator can be calculated recursively via Algorithm 1. The main difference from [ZCW20] is the use of the stochastic zeroth-order gradient, due to which, several assumptions made in [ZCW20] are not satisfied in the stochastic zeroth-order setting we consider. Theorem 3.1 shows that the recursive estimator Σ n converges in operator norm to the asymptotic covariance matrix Ṽ appearing in Proposition 2.1.…”

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

“…We now provide a sketch of the proof highlighting the main differences from the proof of [ZCW20]. We defer the full proof to Section C, as it is involved.…”

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

“…We could now adapt the proof of Theorem 1 in [ZCW20] by replacing martingale difference ξn in first-order SGD iterates by the new difference sequence ξn = ∇f (x n−1 ) − ∇ F x n−1 , ξn .…”

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

See 4 more Smart Citations

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

Jin¹,

Xiao²,

Balasubramanian³

2021

Preprint

View full text Add to dashboard Cite

As machine learning models are deployed in critical applications, it becomes important to not just provide point estimators of the model parameters (or subsequent predictions), but also quantify the uncertainty associated with estimating the model parameters via confidence sets. In the last decade, estimating or training in several machine learning models has become synonymous with running stochastic gradient algorithms. However, computing the stochastic gradients in several settings is highly expensive or even impossible at times. An important question which has thus far not been addressed sufficiently in the statistical machine learning literature is that of equipping zeroth-order stochastic gradient algorithms with practical yet rigorous inferential capabilities. Towards this, in this work, we first establish a central limit theorem for Polyak-Ruppert averaged stochastic gradient algorithm in the zeroth-order setting. We then provide online estimators of the asymptotic covariance matrix appearing in the central limit theorem, thereby providing a practical procedure for constructing asymptotically valid confidence sets (or intervals) for parameter estimation (or prediction) in the zeroth-order setting.

show abstract

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

“…We now provide a sketch of the proof highlighting the main differences from the proof of [ZCW20]. We defer the full proof to Section C, as it is involved.…”

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

Section: Online Estimation Of Asymptotic Covariance Matrixmentioning

confidence: 99%

See 3 more Smart Citations

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

Jin¹,

Xiao²,

Balasubramanian³

2021

Preprint

View full text Add to dashboard Cite

show abstract

A sieve stochastic gradient descent estimator for online nonparametric regression in Sobolev ellipsoids

Zhang

Simon

2022

Ann. Statist.

View full text Add to dashboard Cite

In this paper, we investigate the theoretical properties of stochastic gradient descent (SGD) for statistical inference in the context of nonconvex optimization problems, which have been relatively unexplored compared to convex settings. Our study is the first to establish provable inferential procedures using the SGD estimator for general nonconvex objective functions, which may contain multiple local minima.We propose two novel online inferential procedures that combine SGD and the multiplier bootstrap technique. The first procedure employs a consistent covariance matrix estimator, and we establish its error convergence rate. The second procedure approximates the limit distribution using bootstrap SGD estimators, yielding asymptotically valid bootstrap confidence intervals. We validate the effectiveness of both approaches through numerical experiments.Furthermore, our analysis yields an intermediate result: the in-expectation error convergence rate for the original SGD estimator in nonconvex settings, which is comparable to existing results for convex problems. We believe this novel finding holds independent interest and enriches the literature on optimization and statistical inference.

show abstract

Online Covariance Matrix Estimation in Stochastic Gradient Descent

Cited by 2 publications

References 36 publications

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

A sieve stochastic gradient descent estimator for online nonparametric regression in Sobolev ellipsoids

Contact Info

Product

Resources

About