POLA: Online Time Series Prediction by Adaptive Learning Rates

Abstract

Online prediction for streaming time series data has practical use for many real-world applications where downstream decisions depend on accurate forecasts for the future. Deployment in dynamic environments requires models to adapt quickly to changing data distributions without overfitting. We propose POLA (Predicting Online by Learning rate Adaptation) to automatically regulate the learning rate of recurrent neural network models to adapt to changing time series patterns across time. POLA meta-learns the learning rate of the stochastic gradient descent (SGD) algorithm by assimilating the prequential or interleaved-test-then-train evaluation scheme for online prediction. We evaluate POLA on two real-world datasets across three commonly-used recurrent neural network models. POLA demonstrates overall comparable or better predictive performance over other online prediction methods.

Online Attention Enhanced Differential and Decomposed LSTM for Time Series Prediction

“…Liu et al [5] initiate model updates for online prediction after detecting data distribution changes through concept drift detection and specified rules. Zhang [23] adopts meta-learning to adaptively adjust the learning rate of the stochastic gradient descent (SGD) in recurrent neural networks to continuously fit time series data. These two schemes either introduce additional processing steps or are only applicable to the network structure with SGD, and the serious prediction lag still exists in some cases.…”

“…the predicted data trend lags behind the actual trend). Currently, the accurate and lag-free prediction of time series has become an important and challenging issue in data analysis [22], [23].…”

“…In view of the advantages of memory, hyper-parameter sharing and context awareness, variations of recursive neural network (RNN), long and short term memory (LSTM) and Transformer have been applied to time series data prediction [30]- [32], and the prediction performance is improved to a certain extent. Moreover, online learning mode and data decomposition method [33] have been introduced in some deep learning [23], [31], to adapt to the changes of time series patterns over time, resulting in improved training time and prediction performance of the models. Therefore, the deep learning model has gradually become the mainstream model of time series data prediction.…”

Bearing Fault Identification Method under Small Samples and Multiple Working Conditions

“…Meta-SGD can learn task-agnostic features rather than simply adapt to task-speci c features [28]. Literature [29] used the learning rate of the meta-learning SGD to predict streaming time-series data online and achieved good results. [30] is a multimodel integration method, and its nal output is the weighted sum of the results of the integrated multiple classi ers.…”