Position-Based Content Attention for Time Series Forecasting with Sequence-to-Sequence RNNs

Cinar, Yagmur Gizem; Mirisaee, Hamid; Goswami, Parantapa; Gaussier, Éric; Aït-Bachir, Ali; Strijov, Vadim

doi:10.1007/978-3-319-70139-4_54

Cited by 57 publications

(32 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although the BL learns separate dependencies along each mode, it is not obvious how a representation at a time instance interacts with other time instances or which time instances are important to the prediction at horizon T ′ . By incorporating the position information into the attention calculation scheme, the authors in [56] showed that the learned model only used a particular time instance in the past sequence to predict the future value at a given horizon for sequence-to-sequence learning. In order to learn the importance of each time instance in the proposed BL, we propose the Temporal Attention augmented Bilinear Layer (TABL) that maps the input X ∈ R D×T to the output Y ∈ R D ′ ×T ′ as follows:…”

Section: B Temporal Attention Augmented Bilinear Layermentioning

confidence: 99%

Temporal Attention-Augmented Bilinear Network for Financial Time-Series Data Analysis

Tran

Iosifidis

Kanniainen

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

198

117

View full text Add to dashboard Cite

Financial time-series forecasting has long been a challenging problem because of the inherently noisy and stochastic nature of the market. In the High-Frequency Trading (HFT), forecasting for trading purposes is even a more challenging task since an automated inference system is required to be both accurate and fast. In this paper, we propose a neural network layer architecture that incorporates the idea of bilinear projection as well as an attention mechanism that enables the layer to detect and focus on crucial temporal information. The resulting network is highly interpretable, given its ability to highlight the importance and contribution of each temporal instance, thus allowing further analysis on the time instances of interest. Our experiments in a large-scale Limit Order Book (LOB) dataset show that a two-hidden-layer network utilizing our proposed layer outperforms by a large margin all existing state-of-the-art results coming from much deeper architectures while requiring far fewer computations.

show abstract

Section: B Temporal Attention Augmented Bilinear Layermentioning

confidence: 99%

Temporal Attention-Augmented Bilinear Network for Financial Time-Series Data Analysis

Tran

Iosifidis

Kanniainen

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

198

117

View full text Add to dashboard Cite

show abstract

“…The attention mechanism is typically used in RNN architectures to improve the model performance. The works in [65][66][67] are a few examples where attention blocks were proposed and used with LSTM architectures for time series forecasting. Attention blocks have also been used with CNN architectures [40,43,68,69] for image classification and time series data.…”

Section: The Proposed Modelmentioning

confidence: 99%

Multi-Step Short-Term Wind Speed Prediction Using a Residual Dilated Causal Convolutional Network with Nonlinear Attention

Kumar

Tzou

2020

Energies

View full text Add to dashboard Cite

Wind energy is the most used renewable energy worldwide second only to hydropower. However, the stochastic nature of wind speed makes it harder for wind farms to manage the future power production and maintenance schedules efficiently. Many wind speed prediction models exist that focus on advance neural networks and/or preprocessing techniques to improve the accuracy. Since most of these models require a large amount of historic wind data and are validated using the data split method, the application to real-world scenarios cannot be determined. In this paper, we present a multi-step univariate prediction model for wind speed data inspired by the residual U-net architecture of the convolutional neural network (CNN). We propose a residual dilated causal convolutional neural network (Res-DCCNN) with nonlinear attention for multi-step-ahead wind speed forecasting. Our model can outperform long-term short-term memory networks (LSTM), gated recurrent units (GRU), and Res-DCCNN using sliding window validation techniques for 50-step-ahead wind speed prediction. We tested the performance of the proposed model on six real-world wind speed datasets with different probability distributions to confirm its effectiveness, and using several error metrics, we demonstrated that our proposed model was robust, precise, and applicable to real-world cases.

show abstract

“…Bianchi et al 2017compared different RNN variants and showed that LSTMs outperformed others on highly non-linear sequences with sharp spikes thanks to the quick memory cell modification mechanism. Cinar et al (2017) proposed using an LSTM encoder-decoder with position-based attention model to capture patterns of pseudo-periods in sequence data. They applied the attention mechanism (Bahdanau et al 2014) to explore similar local patterns in historical data for future prediction.…”

Section: Forecasting With Deep Learning Techniquesmentioning

confidence: 99%

Retail supply chain management: a review of theories and practices

Ge¹,

Pan²,

Shen

et al. 2019

J. of Data, Inf. and Manag.

View full text Add to dashboard Cite

Retail business has been rapidly evolving in the past decades with the boom of internet, mobile technologies and most importantly e-commerce. Supply chain management, as a core part of retail business, has also gone through significant changes with new business scenarios and more advanced technologies in both algorithm design and computation power. In this review, we focus on several core components of supply chain management, i.e. vendor management, demand forecasting, inventory management and order fulfillment. We will discuss the key innovations from both academia and industry and highlight the current trend and future challenges.

show abstract

Position-Based Content Attention for Time Series Forecasting with Sequence-to-Sequence RNNs

Cited by 57 publications

References 25 publications

Temporal Attention-Augmented Bilinear Network for Financial Time-Series Data Analysis

Temporal Attention-Augmented Bilinear Network for Financial Time-Series Data Analysis

Multi-Step Short-Term Wind Speed Prediction Using a Residual Dilated Causal Convolutional Network with Nonlinear Attention

Retail supply chain management: a review of theories and practices

Contact Info

Product

Resources

About