Temporal Convolutional Attention-based Network For Sequence Modeling

Hao, Hongyan; Wang, Yan; Xia, Yudi; Zhao, Jian; Shen, Furao

doi:10.48550/arxiv.2002.12530

Cited by 13 publications

(20 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, deep learning models [31,38] are rising as the new state-of-the-arts for many time series forecasting tasks. Many of them use the sequenceto-sequence neural structures [13,48] based on recurrent neural networks [20,26,43], Convolutional neural networks [2,18], transformers [29,53,57], and graph neural networks [54]. Deep learning models are also efficient in modeling multivariate time series [59].…”

Section: Time Series Forecastingmentioning

confidence: 99%

Learning Differential Operators for Interpretable Time Series Modeling

Luo

Liu

et al. 2022

Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Modeling sequential patterns from data is at the core of various time series forecasting tasks. Deep learning models have greatly outperformed many traditional models, but these black-box models generally lack explainability in prediction and decision making. To reveal the underlying trend with understandable mathematical expressions, scientists and economists tend to use partial differential equations (PDEs) to explain the highly nonlinear dynamics of sequential patterns. However, it usually requires domain expert knowledge and a series of simplified assumptions, which is not always practical and can deviate from the ever-changing world. Is it possible to learn the differential relations from data dynamically to explain the time-evolving dynamics? In this work, we propose an learning framework that can automatically obtain interpretable PDE models from sequential data. Particularly, this framework is comprised of learnable differential blocks, named 𝑃-blocks, which is proved to be able to approximate any time-evolving complex continuous functions in theory. Moreover, to capture the dynamics shift, this framework introduces a meta-learning controller to dynamically optimize the hyper-parameters of a hybrid PDE model. Extensive experiments on times series forecasting of financial, engineering, and health data show that our model can provide valuable interpretability and achieve comparable performance to state-ofthe-art models. From empirical studies, we find that learning a few differential operators may capture the major trend of sequential dynamics without massive computational complexity. CCS CONCEPTS• Computing methodologies → Machine learning algorithms;• Information systems → Data mining.

show abstract

Section: Time Series Forecastingmentioning

confidence: 99%

Learning Differential Operators for Interpretable Time Series Modeling

Luo

Liu

et al. 2022

Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

show abstract

“…Our explorations into the domain of sequence modeling to generate these forecasts, i.e., time-series predictions, have taken us from statistical engines to multi-layer perceptrons (MLPs) to recursive models [26], [27].…”

Section: Related Workmentioning

confidence: 99%

Parallel spatio-temporal attention-based TCN for multivariate time series prediction

Fan

Zhang

Huang

et al. 2021

Neural Comput & Applic

113

View full text Add to dashboard Cite

As industrial systems become more complex and monitoring sensors for everything from surveillance to our health become more ubiquitous, multivariate time series prediction is taking an important place in the smooth-running of our society. A recurrent neural network with attention to help extend the prediction windows is the current-state-of-the-art for this task. However, we argue that their vanishing gradients, short memories, and serial architecture make RNNs fundamentally unsuited to long-horizon forecasting with complex data. Temporal convolutional networks (TCNs) do not suffer from gradient problems and they support parallel calculations, making them a more appropriate choice. Additionally, they have longer memories than RNNs, albeit with some instability and efficiency problems. Hence, we propose a framework, called PSTA-TCN, that combines a parallel spatio-temporal attention mechanism to extract dynamic internal correlations with stacked TCN backbones to extract features from different window sizes. The framework makes full use parallel calculations to dramatically reduce training times, while substantially increasing accuracy with stable prediction windows up to 13 times longer than the status quo.

show abstract

“…Most successful neural networks are based on the encoder-decoder architectures [36,10,12,33,3,11,20,21], namely Seq2seq. Basically, various Seq2seq models based on RNNs [16,32,37,18,41,22], CNNs [4,15], and Transformers (self-attentions) [19,38] are proposed to model the non-linearity for time series.…”

Section: Related Workmentioning

confidence: 99%

“…We show that as we can identify the ground truth components of the mixture, and the ground truth assignment of each microscopic observation, independent modeling of time series data from each component could be improved due to lower variance, and further benefitting the estimation of macroscopic time series that are of interest. Second, inspired by recent successes of Seq2seq models [36,10,12] based on deep neural networks, e.g., variants of recurrent neural networks (RNNs) [16,41,22], convolutional neural networks (CNNs) [4,15], and Transformers [19,38], we propose Mixture of Seq2seq (MixSeq), a mixture model for time series, where the components come from a family of Seq2seq models parameterized by different parameters. Third, we conduct synthetic experiments to demonstrate the superiority of our approach, and extensive experiments on real-world data to show the power of our approach compared with canonical approaches.…”

Section: Introductionmentioning

confidence: 99%

MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data

Zhu¹,

Liu²,

Jin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Time series forecasting is widely used in business intelligence, e.g., forecast stock market price, sales, and help the analysis of data trend. Most time series of interest are macroscopic time series that are aggregated from microscopic data. However, instead of directly modeling the macroscopic time series, rare literature studied the forecasting of macroscopic time series by leveraging data on the microscopic level. In this paper, we assume that the microscopic time series follow some unknown mixture probabilistic distributions. We theoretically show that as we identify the ground truth latent mixture components, the estimation of time series from each component could be improved because of lower variance, thus benefitting the estimation of macroscopic time series as well. Inspired by the power of Seq2seq and its variants on the modeling of time series data, we propose Mixture of Seq2seq (MixSeq), an end2end mixture model to cluster microscopic time series, where all the components come from a family of Seq2seq models parameterized by different parameters. Extensive experiments on both synthetic and real-world data show the superiority of our approach.

show abstract

Temporal Convolutional Attention-based Network For Sequence Modeling

Cited by 13 publications

References 6 publications

Learning Differential Operators for Interpretable Time Series Modeling

Learning Differential Operators for Interpretable Time Series Modeling

Parallel spatio-temporal attention-based TCN for multivariate time series prediction

MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data

Contact Info

Product

Resources

About