Adaptive regularization for Lasso models in the context of nonstationary data streams

Abstract

Large-scale, streaming data sets are ubiquitous in modern machine learning. Streaming algorithms must be scalable, amenable to incremental training, and robust to the presence of nonstationarity. In this work we consider the problem of learning 1 regularized linear models in the context of streaming data. In particular, the focus of this work revolves around how to select the regularization parameter when data arrives sequentially and the underlying distribution is nonstationary (implying the choice of optimal regularization parameter is itself time-varying). We propose a framework through which to infer an adaptive regularization parameter. Our approach employs an 1 penalty constraint where the corresponding sparsity parameter is iteratively updated via stochastic gradient descent. This serves to reformulate the choice of regularization parameter in a principled framework for online learning. The proposed method is derived for linear regression and subsequently extended to generalized linear models. We validate our approach using simulated and real data sets, concluding with an application to a neuroimaging data set.

Estimation, Forecasting, and Anomaly Detection for Nonstationary Streams Using Adaptive Estimation

“…The contribution of this paper is the development of a forecasting procedure and associated local anomaly detector, capable of dealing with the many challenges of streaming data. This is achieved by extending the results of [31]. First, proposing two new adaptive estimators for a penalized-regression model.…”

“…Other domains exhibit data with more complicated, or random arrival order, leading to the delayed labelling problem [35]. To achieve this, we require FF estimates of both mean and covariance of response y t and basis vector x t at each tick t. In [15], [17], [31], an adaptive estimation framework was used for both sample mean vector, and sample covariance matrix. Define Π t = y t , x…”

“…The effect of these initial values will vanish when adopting a burn-in period of B observations. Similar to the sequential algorithms proposed in [15], [17], [31], assuming that observations y 1 , ...y B will not face any change in the underlying structure, this set of observations will be used to estimate the initial values.…”

“…These bases are potentially overparametrized and hence streaming penalization methods, adapted from procedures such as LASSO, are required. In this case, the choice of an optimal regularization parameter may itself be time-varying [31]. In order to sequentially fit the underlying structure of y t , a time-varying penalty parameter γ t ∈ R + is introduced (c.f.…”

“…By default, these models are fitted using a grid of penalty parameters which are computationally unfeasible in streaming data context. Similarly, [31] proposed an updating rule to the penalty term designed to estimate the parameters as soon as new data arrives and assuming the underlying distribution is nonstationary. Similar to AFF, the framework of [31] also adopts adaptive filtering estimation based on stochastic gradient descent.…”