Transformers in Time-Series Analysis: A Tutorial

Ahmed, Sabeen; Nielsen, Ian E.; Tripathi, Aakash; Siddiqui, Sahabjada; Ramachandran, Ravi P.; Rasool, Ghulam

doi:10.1007/s00034-023-02454-8

Cited by 68 publications

(10 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The decoder then condensed this representation into the target output sequence, using masked self-attention to avoid information leakage. Modeling of long-range temporal relationships was enabled by attention in the data; nonlinear feature transformation was executed by the feedforward layers [ 20 ].…”

Section: Methodsmentioning

confidence: 99%

Estimating Rotational Acceleration in Shoulder and Elbow Joints Using a Transformer Algorithm and a Fusion of Biosignals

Bai,

Guan,

et al. 2024

Sensors

View full text Add to dashboard Cite

In the present study, we used a transformer model and a fusion of biosignals to estimate rotational acceleration in elbow and shoulder joints. To achieve our study objectives, we proposed a mechanomyography (MMG) signal isolation technique based on a variational mode decomposition (VMD) algorithm. Our results show that the VMD algorithm delivered excellent performance in MMG signal extraction compared to the commonly used technique of empirical mode decomposition (EMD). In addition, we found that transformer models delivered estimates of joint acceleration that were more precise than those produced by mainstream time series forecasting models. The average R2 values of transformer are 0.967, 0.968, and 0.935, respectively. Finally, we found that using a fusion of signals resulted in more precise estimation performance compared to using MMG signals alone. The differences between the average R2 values are 0.041, 0.053, and 0.043, respectively. Taken together, the VMD isolation method, the transformer algorithm and the signal fusion technique described in this paper can be seen as supplying a robust framework for estimating rotational acceleration in upper-limb joints. Further study is warranted to examine the effectiveness of this framework in other musculoskeletal contexts.

show abstract

Section: Methodsmentioning

confidence: 99%

Estimating Rotational Acceleration in Shoulder and Elbow Joints Using a Transformer Algorithm and a Fusion of Biosignals

Bai,

Guan,

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…The overall architecture of the transformer model is depicted in Figure 4. The transformer model mainly consists of input, encoder block, decoder block, and output components, among which the encoder and decoder blocks are two critical parts [62]. First, we denote the time-series images as X = {x 1 , x 2 , .…”

Section: Transformer Modelmentioning

confidence: 99%

A Transformer Model for Coastline Prediction in Weitou Bay, China

Yang,

Wang,

Feng

et al. 2023

Remote Sensing

View full text Add to dashboard Cite

The simulation and prediction of coastline changes are of great significance for the development and scientific management of coastal zones. Coastline changes are difficult to capture completely but appear significantly periodic over a long time series. In this paper, the transformer model is used to learn the changing trend of the coastline so as to deduce the position of the coastline in the coming year. First, we use the distance regularization level set evolution (DRLSE) model for instantaneous waterline extraction (IWE) from preprocessed Landsat time-series images from 2010–2020 in Weitou Bay, China. Then, tidal correction (TC) is performed on the extracted instantaneous waterline dataset to obtain coastlines projected to a single reference tidal datum. Finally, the coastline datasets from 2010–2019 are used for model training, and the coastline in 2020 is used for accuracy assessment. Three precision evaluation methods, including receiver operating characteristic curve matching, the mean offset, and the root mean square error, were used to verify the predicted coastline data. The receiver operating characteristic curve was specifically designed and improved to evaluate the accuracy of the obtained coastline. Compared with the support vector regression (SVR) and long–short-term memory (LSTM) methods, the results showed that the coastline predicted by the transformer model was the closest to the accurate extracted coastline. The accuracies of the correct values corresponding to SVR, LSTM, and transformer models were 88.27%, 94.08%, and 98.80%, respectively, which indicated the accuracy of the coastline extraction results. Additionally, the mean offset and root mean square error were 0.32 pixels and 0.57 pixels, respectively. In addition, the experimental results showed that tidal correction is important for coastline prediction. Moreover, through field investigations of coastlines, the predicted results obtained for natural coastlines were more accurate, while the predicted results were relatively poor for some artificial coastlines that were intensely influenced by human activities. This study shows that the transformer model can provide natural coastline changes for coastal management.

show abstract

“…Transformers are deep neural networks that exploit the self-attention mechanism to capture relationships between different portions of a text and have rapidly attracted interest in machine learning. Their applications span several domains [1], including natural language processing, Computer Vision [2], Audio and Speech signals [3], and Signal Processing [4]. Their popularity is making transformers become one of the most used deep learning architectures.…”

Section: Introductionmentioning

confidence: 99%

The Explainability of Transformers: Current Status and Directions

Fantozzi,

Naldi

2024

Computers

View full text Add to dashboard Cite

An increasing demand for model explainability has accompanied the widespread adoption of transformers in various fields of applications. In this paper, we conduct a survey of the existing literature on the explainability of transformers. We provide a taxonomy of methods based on the combination of transformer components that are leveraged to arrive at the explanation. For each method, we describe its mechanism and survey its applications. We find out that attention-based methods, both alone and in conjunction with activation-based and gradient-based methods, are the most employed ones. A growing attention is also devoted to the deployment of visualization techniques to help the explanation process.

show abstract

Transformers in Time-Series Analysis: A Tutorial

Cited by 68 publications

References 58 publications

Estimating Rotational Acceleration in Shoulder and Elbow Joints Using a Transformer Algorithm and a Fusion of Biosignals

Estimating Rotational Acceleration in Shoulder and Elbow Joints Using a Transformer Algorithm and a Fusion of Biosignals

A Transformer Model for Coastline Prediction in Weitou Bay, China

The Explainability of Transformers: Current Status and Directions

Contact Info

Product

Resources

About