Cogra: Concept-Drift-Aware Stochastic Gradient Descent for Time-Series Forecasting

Abstract

We approach the time-series forecasting problem in the presence of concept drift by automatic learning rate tuning of stochastic gradient descent (SGD). The SGD-based approach is preferable to other concept drift algorithms in that it can be applied to any model and it can keep learning efficiently whilst predicting online. Among a number of SGD algorithms, the variance-based SGD (vSGD) can successfully handle concept drift by automatic learning rate tuning, which is reduced to an adaptive mean estimation problem. However, its performance is still limited because of its heuristic mean estimator. In this paper, we present a concept-drift-aware stochastic gradient descent (Cogra), equipped with more theoretically-sound mean estimator called sequential mean tracker (SMT). Our key contribution is that we define a goodness criterion for the mean estimators; SMT is designed to be optimal according to this criterion. As a result of comprehensive experiments, we find that (i) our SMT can estimate the mean better than vSGD's estimator in the presence of concept drift, and (ii) in terms of predictive performance, Cogra reduces the predictive loss by 16-67% for real-world datasets, indicating that SMT improves the prediction accuracy significantly.

We got the power: Predicting available capacity for vehicle-to-grid services using a deep recurrent neural network

“…Training data was presented to the model in batches of 48 timesteps, which is representative of a real-life scenario in which 24 hours of data may be received at the end of each day's operation and used to continually update the model. This process would allow refinement of the model to variation in the fleet's behaviour, however, more substantial changes may require concept drift aware algorithms that automatically tune the learning rate to support more rapid adjustment of network parameters [38].…”

POLA: Online Time Series Prediction by Adaptive Learning Rates

“…Gradient descent algorithms are commonly used to train neural network models online by constantly tracking a specified performance measure such as prediction error, and are also applied to traditional statistical methods such as the autoregressive integrated moving average models [8,16]. The learning rate can be adapted to achieve better convergence, avoid overfitting to noisy samples or to automatically adjust to shifts in data distribution [5,14,17,18,19]. RM-Sprop [20] is a popular algorithm that scales the learning rate by moving average of squared gradients based on the intuition that the magnitude of each weight update should be similar regardless of the actual gradient magnitude.…”

“…In POLA, instead of explicitly applying an inductive bias into learning rate scaling, we propose learning this scaling directly from the data itself. For adaptation in non-stationary environments, previous works apply simplifying assumptions on the loss function and Hessian computations [5,14], however secondorder derivative calculation can be expensive and the resulting learning rates are subject to instability if the approximated Hessian is ill-conditioned. Other works adapt the learning rate to reduce the adverse effect of outliers by monitoring distributional properties at neighboring data points [21,22].…”

“…In this work, we focus on online time series prediction which is studied and applied in a wide range of domains such as in traffic monitoring, climate research and finance [5,6,7,8]. Predictions can be used for downstream tasks such as anomaly detection and to aid decision making [9,10].…”

“…Most works design algorithms for stationary environments with i.i.d. samples [4], and others that do consider time series prediction require Hessian computations [5,14] Fig. 1: Overview: POLA adapts the learning rate of SGD update through factor βt`1 P r0, 1s where the factor is meta-learned with the most recent data batch by assimilating online evaluation.…”