A non-parametric softmax for improving neural attention in time-series forecasting

Totaro, Simone; Hussain, Amir; Scardapane, Simone

doi:10.1016/j.neucom.2019.10.084

Cited by 24 publications

(18 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequently, scholars have further refined it under its structures. Toraro et al 47 study a learnable softmax operation to replace the regular softmax layer on top of DA‐RNN. Hu and Zheng 48 also improved on DA‐RNN with an LSTM model and a first‐stage attentional model as an encoder to make predictions about MTS data.…”

Section: Related Workmentioning

confidence: 99%

A dual‐stage attention‐based Conv‐LSTM network for spatio‐temporal correlation and multivariate time series prediction

et al. 2021

View full text Add to dashboard Cite

Multivariate time series (MTS) prediction aims at predicting future time series by extracting multiple forms of dependencies of past time series. Traditional prediction methods and deep learning‐based prediction methods focus on extracting the dynamic relationships of certain aspects of MTS, especially the temporal characteristics, often neglecting the spatial and temporal dynamic correlations of MTS. Inspired by convolution neural network (CNN) and attention mechanism, this paper proposes a convolution LSTM network model based on MTS prediction with two‐stage attention. Specifically, we first propose a new MTS preprocessing method to perform convolution operations better. Then convolution layer extracts spatial correlation of MTS and LSTM model extracts temporal correlation. It is worth mentioning that the combination of attention mechanism and LSTM can effectively solve the problem of insufficient time dependency in MTS prediction. In addition, dual‐stage attention mechanism can effectively eliminate irrelevant information, select the relevant exogenous sequence, give it higher weight, and increase the past value of the target sequence to further eliminate irrelevant information. Finally, the MTS spatio‐temporal correlation is extracted to improve the prediction accuracy, and the model is interpreted. Experimental results show that the model has broad application prospects. Experiments based on typical datasets of finance, environment, and energy determine the optimal window size and hidden size of the prediction, and demonstrate that the model achieves the state‐of‐the‐art effect compared to the other four deep learning models. On top of that, the model is not only suitable for single‐step prediction of MTS, but also suitable for multistep prediction of time step in a certain range.

show abstract

Section: Related Workmentioning

confidence: 99%

A dual‐stage attention‐based Conv‐LSTM network for spatio‐temporal correlation and multivariate time series prediction

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Neural networks are trained using Softmax as an activation function [31]. Softmax has the advantage of being computationally convenient and having a simple form of results for multiple classification problems.…”

Section: Softmax Activation Functionmentioning

confidence: 99%

Music Recognition and Classification Algorithm considering Audio Emotion

Wang

Fang

2022

Scientific Programming

View full text Add to dashboard Cite

At present, the existing music classification and recognition algorithms have the problem of low accuracy. Therefore, this paper proposes a music recognition and classification algorithm considering the characteristics of audio emotion. Firstly, the emotional features of music are extracted from the feedforward neural network and parameterized with the mean square deviation. Gradient descent learning algorithm is used to train audio emotion features. The neural network models of input layer, output layer, and hidden layer are established to realize the classification and recognition of music emotion. Experimental results show that the algorithm has good effect on music emotion classification. The data stream driven by the algorithm is higher than 55 MBbs, the anti-attack ability is 91%, the data integrity is 83%, the average accuracy is 85%, and it has good effectiveness and feasibility.

show abstract

“…This feature is particularly beneficial in time series prediction as it allows the model to weigh the importance of past and recent events differently, thereby significantly improving the accuracy of the predictions. The authors in (Totaro, Hussain and Scardapane, 2020) demonstrate the application of this technique in time series forecasting with the dual-stage attention-based recurrent neural network. This model is applied to the hourly data of Dogecoin price for its prediction over time.…”

Section: ) Related Workmentioning

confidence: 99%

Tuning of Novel Fractional Order Fuzzy PID Controller for Automatic Voltage Regulator using Grasshopper Optimization Algorithm

Khaniki

Hadi

Manthouri

2021

mjee

View full text Add to dashboard Cite

One of the essential pieces of equipment in the power system is the Automatic Voltage Regulator (AVR) or synchronous generator excitation. The system's goal is to maintain the terminal voltage of the synchronous generator at the desired level. AVR is inherently uncertain. Hence, the proposed controller should be able to handle the problem. In this paper, Fractional Order Fuzzy PID (FOFPID) controller has been employed to control the system. To enhance the controller's performance, the Grasshopper Optimization Algorithm (GOA) is used to tune the controller's parameters. Unlike other methods, the FOFPID controller gains are not constant and alter in different operating conditions. The robustness of the controller has been investigated, and the comparative results show that the proposed controller has a better performance against other methods.

show abstract

A non-parametric softmax for improving neural attention in time-series forecasting

Cited by 24 publications

References 36 publications

A dual‐stage attention‐based Conv‐LSTM network for spatio‐temporal correlation and multivariate time series prediction

A dual‐stage attention‐based Conv‐LSTM network for spatio‐temporal correlation and multivariate time series prediction

Music Recognition and Classification Algorithm considering Audio Emotion

Tuning of Novel Fractional Order Fuzzy PID Controller for Automatic Voltage Regulator using Grasshopper Optimization Algorithm

Contact Info

Product

Resources

About