Time Series is a Special Sequence: Forecasting with Sample Convolution and Interaction

Liu, Minhao; Zeng, Ailing; Lai, Qiuxia; Xu, Qiang

doi:10.48550/arxiv.2106.09305

Cited by 29 publications

(49 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RNN-based TSF methods belong to IMS forecasting techniques. Depending on whether the decoder is implemented in an autoregressive manner, there are either IMS or DMS forecasting techniques for CNN-based TSF methods [3,17].…”

Section: Non-transformer-based Tsf Solutionsmentioning

confidence: 99%

Are Transformers Effective for Time Series Forecasting?

Zeng¹,

Chen²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, there has been a surge of Transformer-based solutions for the time series forecasting (TSF) task, especially for the challenging long-term TSF problem. Transformer architecture relies on self-attention mechanisms to effectively extract the semantic correlations between paired elements in a long sequence, which is permutation-invariant and "anti-ordering" to some extent. However, in time series modeling, we are to extract the temporal relations among an ordering set of continuous points. Consequently, whether Transformer-based techniques are the right solutions for "long-term" time series forecasting is an interesting problem to investigate, despite the performance improvements shown in these studies. In this work, we question the validity of Transformer-based TSF solutions. In their experiments, the compared (non-Transformer) baselines are mainly autoregressive forecasting solutions, which usually have a poor long-term prediction capability due to inevitable error accumulation effects. In contrast, we use an embarrassingly simple architecture named DLinear that conducts direct multi-step (DMS) forecasting for comparison. DLinear decomposes the time series into a trend and a remainder series and employs two one-layer linear networks to model these two series for the forecasting task. Surprisingly, it outperforms existing complex Transformer-based models in most cases by a large margin. Therefore, we conclude that the relatively higher long-term forecasting accuracy of Transformer-based TSF solutions shown in existing works has little to do with the temporal relation extraction capabilities of the Transformer architecture. Instead, it is mainly due to the non-autoregressive DMS forecasting strategy used in them. We hope this study also advocates revisiting the validity of Transformer-based solutions for other time series analysis tasks (e.g., anomaly detection) in the future. Code is available at https://github.com/cure-lab/DLinear. * Equal contribution Preprint. Under review.

show abstract

Section: Non-transformer-based Tsf Solutionsmentioning

confidence: 99%

Are Transformers Effective for Time Series Forecasting?

Zeng¹,

Chen²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recent architectures like the Reformer [21], LinFormer [50] and Informer [54] focused on reducting this cost by introducing restricted attention layers to effectively approximate the full attention mechanism. Currently, the best performing model architectures are SCINet [25] and N-BEATS [36] on all common datasets and we will compare against them as baselines.…”

Section: Related Workmentioning

confidence: 99%

Few-Shot Forecasting of Time-Series with Heterogeneous Channels

Brinkmeyer¹,

Drumond²,

Burchert³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning complex time series forecasting models usually requires a large amount of data, as each model is trained from scratch for each task/data set. Leveraging learning experience with similar datasets is a well-established technique for classification problems called few-shot classification. However, existing approaches cannot be applied to timeseries forecasting because i) multivariate time-series datasets have different channels and ii) forecasting is principally different from classification. In this paper we formalize the problem of few-shot forecasting of timeseries with heterogeneous channels for the first time. Extending recent work on heterogeneous attributes in vector data, we develop a model composed of permutation-invariant deep set-blocks which incorporate a temporal embedding. We assemble the first meta-dataset of 40 multivariate time-series datasets and show through experiments that our model provides a good generalization, outperforming baselines carried over from simpler scenarios that either fail to learn across tasks or miss temporal information.

show abstract

“…Deep learning (DL) is commonly used in large data analysis to tackle a variety of issues, such as object recognition [8,9], speech recognition [10], classification [11], and prediction based on time series data [12]. The forecasted accuracy and efficiency of air quality is enhanced with the use of deep learning approaches.…”

Section: Introductionmentioning

confidence: 99%

Attention-Based Distributed Deep Learning Model for Air Quality Forecasting

et al. 2022

View full text Add to dashboard Cite

Air quality forecasting has become an essential factor in facilitating sustainable development worldwide. Several countries have implemented monitoring stations to collect air pollution particle data and meteorological information using parameters such as hourly timespans. This research focuses on unravelling a new framework for air quality prediction worldwide and features Busan, South Korea as its model city. The paper proposes the application of an attention-based convolutional BiLSTM autoencoder model. The proposed deep learning model has been trained on a distributed framework, referred to data parallelism, to forecast the intensity of particle pollution ( and ). The algorithm automatically learns the intrinsic correlation among the particle pollution in different locations. Each location’s meteorological and traffic data is extensively exploited to improve the model’s performance. The model has been trained using air quality particle data and car traffic information. The traffic information is obtained by a device which counts cars passing a specific area through the YOLO algorithm, and then sends the data to a stacked deep autoencoder to be encoded alongside the meteorological data before the final prediction. In addition, multiple one-dimensional CNN layers are used to obtain the local spatial features jointly with a stacked attention-based BiLSTM layer to figure out how air quality particles are correlated in space and time. The evaluation of the new attention-based convolutional BiLSTM autoencoder model was derived from data collected and retrieved from comprehensive experiments conducted in South Korea. The results not only show that the framework outperforms the previous models both on short- and long-term predictions but also indicate that traffic information can improve the accuracy of air quality forecasting. For instance, during prediction, the proposed attention-based model obtained the lowest MAE (5.02 and 22.59, respectively, for short-term and long-term prediction), RMSE (7.48 and 28.02) and SMAPE (17.98 and 39.81) among all the models, which indicates strong accuracy between observed and predicted values. It was also found that the newly proposed model had the lowest average training time compared to the baseline algorithms. Furthermore, the proposed framework was successfully deployed in a cloud server in order to provide future air quality information in real time and when needed.

show abstract

Time Series is a Special Sequence: Forecasting with Sample Convolution and Interaction

Cited by 29 publications

References 38 publications

Are Transformers Effective for Time Series Forecasting?

Are Transformers Effective for Time Series Forecasting?

Few-Shot Forecasting of Time-Series with Heterogeneous Channels

Attention-Based Distributed Deep Learning Model for Air Quality Forecasting

Contact Info

Product

Resources

About