Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

Abstract

We develop a new method of online inference for a vector of parameters estimated by the Polyak-Ruppert averaging procedure of stochastic gradient descent (SGD) algorithms. We leverage insights from time series regression in econometrics and construct asymptotically pivotal statistics via random scaling. Our approach is fully operational with online data and is rigorously underpinned by a functional central limit theorem. Our proposed inference method has a couple of key advantages over the existing methods. First, the test statistic is computed in an online fashion with only SGD iterates and the critical values can be obtained without any resampling methods, thereby allowing for efficient implementation suitable for massive online data. Second, there is no need to estimate the asymptotic variance and our inference method is shown to be robust to changes in the tuning parameters for SGD algorithms in simulation experiments with synthetic data.

csa2sls: A complete subset approach for many instruments using Stata

“…Because CSA2SLS is computationally intensive, an interesting future research question would be to develop a more efficient computation algorithm. An approach based on the stochastic gradient descent (see, for example, Lee et al [2022]) can be a possible solution.…”

Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inference

“…The work in Zhu, Chen, and Wu (2023) extends the batch-mean estimator in Chen et al (2020) to a fully online version. The work in Lee et al (2022) proposes a random scaling covariance estimator for robust online inference with SGD. Subsequent work in Li, Liang, and Zhang (2023) investigates online statistical inference using nonlinear SA with Markovian data.…”

“…Recent works have explored using SA and SGD iterates to perform statistical inference, e.g., constructing confidence intervals (CIs) around a point estimate (Li et al 2018;Chen et al 2020;Li et al 2022;Lee et al 2022;Liu, Chen, and Shang 2023). This approach is computationally cheap and scales well with the size and dimension of the dataset: SA updates are computed iteratively, without storing or multiple passes over the dataset.…”

Robust Inference on Infinite and Growing Dimensional Time‐Series Regression

“…For instance, heteroscedasticity and autocorrelation consistent (HAC) estimators, for example, Newey and West ( 1987 ), Andrews ( 1991b ), to name but two examples, have been followed by the fixed‐bandwidth kernel approach to obtain an asymptotically pivotal and mixed‐normal test; see, for example, Kiefer, Vogelsang, and Bunzel ( 2000 ) and Lazarus et al ( 2018 ) for a recent review. In the machine learning literature, Lee, Liao, Seo, and Shin ( 2022 ) also employ the random scaling approach for computationally efficient online inference based on the stochastic gradient descent algorithm.…”