Transformers in Time-series Analysis: A Tutorial

Ahmed, Sabeen; Nielsen, Ian E.; Tripathi, Aakash; Siddiqui, Sahabjada; Rasool, Ghulam; Ramachandran, Ravi P.

doi:10.48550/arxiv.2205.01138

Cited by 5 publications

(6 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ACF graph decays rapidly around zero value when increasing the order of differencing in the stationary dataset; in contrast, ACF graph firstly decreases and then varies periodically in the non-stationary dataset. This means that if the series still exhibits positive ACF at high lags, it may require further differencing [25], [26]. The ACF figures at all other locations (L2, L3, .., L17) produce similar results as that of the firstorder differencing at L1.…”

Section: Proposed DL Model Training and Testing A Data Pre-processingmentioning

confidence: 80%

Deep Learning Models for Time-Series Forecasting of RF-EMF in Wireless Networks

Nguyen,

Cheema,

Kurnaz

et al. 2024

IEEE Open J. Commun. Soc.

View full text Add to dashboard Cite

Radio-frequency electromagnetic field (RF-EMF) forecasting plays an important role in the evaluation of regulatory compliance, network planning and system optimization. The knowledge of RF-EMF levels is essential to ensure compliance with standards and avoid public health concerns, especially with the arrival of new frequencies and scenarios in fifth-generation (5G) and sixth generation (6G) wireless networks. This work provides a comprehensive study on time series forecasting for RF-EMF measured in frequency from 100 kHz -3 GHz. The state-of-the-art deep learning model architectures consist of deep neural network (DNN), convolutional neural network (CNN), long-short term memory (LSTM), and transformer are applied for time series forecasting. The prediction performance is evaluated under three different scenarios -namely single-step input single-step output (SISO), multi-step input single-step output (MISO), and multi-step input multi-step output (MIMO). The findings from the simulation demonstrate that SISO forecasting is inadequate in predicting long-term radio-frequency electromagnetic fields (RF-EMF) data as it lacks accuracy while MISO and MIMO forecasting scenarios offer more precise predictions. Specifically, in these two scenarios where the input width and label width are both set to 20 steps, the LSTM and CNN models exhibit superior performance compared to other models. Nonetheless, as the input width and label width in a MIMO scenario increase, the accuracy of both CNN and LSTM models decline considerably, whereas the transformer model consistently maintains good performance. Additionally, the transformer model continues to deliver accurate predictions as the label width and shift length increase, which is not the case for DNN, CNN, and LSTM models.

show abstract

Section: Proposed DL Model Training and Testing A Data Pre-processingmentioning

confidence: 80%

Deep Learning Models for Time-Series Forecasting of RF-EMF in Wireless Networks

Nguyen,

Cheema,

Kurnaz

et al. 2024

IEEE Open J. Commun. Soc.

View full text Add to dashboard Cite

show abstract

“…The methodologies of the transformer model need to be revisited. For a complete explanation of the algorithm, refer to [22].…”

Section: Soh Estimation Resultsmentioning

confidence: 99%

“…Therefore, j << n. The methodologies of the transformer model need to be revisited. For a complete explanation of the algorithm, refer to [22]. Herein, the loss value used to determine the hyperparameters of the self-attention transformer model can be given by:…”

Section: Soh Estimation Resultsmentioning

confidence: 99%

“…These are manifested as training instabilities leading to vanishing and exploding back-propagated gradient problems [21]. The transformer model, primarily utilized for natural language processing, has recently achieved remarkable advancements in time series forecasting [22]. The transformer model allows for parallel processing, enabling efficient utilization of computing resources and faster training.…”

Section: Literature Review 1modelling and Predicting Battery Statesmentioning

confidence: 99%

See 1 more Smart Citation

Cloud-Based Deep Learning for Co-Estimation of Battery State of Charge and State of Health

et al. 2023

View full text Add to dashboard Cite

Rechargeable lithium-ion batteries are currently the most viable option for energy storage systems in electric vehicle (EV) applications due to their high specific energy, falling costs, and acceptable cycle life. However, accurately predicting the parameters of complex, nonlinear battery systems remains challenging, given diverse aging mechanisms, cell-to-cell variations, and dynamic operating conditions. The states and parameters of batteries are becoming increasingly important in ubiquitous application scenarios, yet our ability to predict cell performance under realistic conditions remains limited. To address the challenge of modelling and predicting the evolution of multiphysics and multiscale battery systems, this study proposes a cloud-based AI-enhanced framework. The framework aims to achieve practical success in the co-estimation of the state of charge (SOC) and state of health (SOH) during the system’s operational lifetime. Self-supervised transformer neural networks offer new opportunities to learn representations of observational data with multiple levels of abstraction and attention mechanisms. Coupling the cloud-edge computing framework with the versatility of deep learning can leverage the predictive ability of exploiting long-range spatio-temporal dependencies across multiple scales.

show abstract

“…This is achieved by integrating positional data into these segments and employing the dot product operation. For a comprehensive understanding of the algorithm and mathematics, please refer to the resource provided in [48]. The proposed Transformer model (Figure 2) consists of four main modules: a dual-embedding module, a two-tower encoder module, sequence predictions, and a gating module.…”

Section: Transformer-based Neural Networkmentioning

confidence: 99%

Spatial-Temporal Self-Attention Transformer Networks for Battery State of Charge Estimation

et al. 2023

View full text Add to dashboard Cite

Over the past ten years, breakthroughs in battery technology have dramatically propelled the evolution of electric vehicle (EV) technologies. For EV applications, accurately estimating the state-of-charge (SOC) is critical for ensuring safe operation and prolonging the lifespan of batteries, particularly under complex loading scenarios. Despite progress in this area, modeling and forecasting the evaluation of multiphysics and multiscale electrochemical systems under realistic conditions using first-principles and atomistic calculations remains challenging. This study proposes a solution by designing a specialized Transformer-based network architecture, called Bidirectional Encoder Representations from Transformers for Batteries (BERTtery), which only uses time-resolved battery data (i.e., current, voltage, and temperature) as an input to estimate SOC. To enhance the Transformer model’s generalization, it was trained and tested under a wide range of working conditions, including diverse aging conditions (ranging from 100% to 80% of the nominal capacity) and varying temperature windows (from 35 °C to −5 °C). To ensure the model’s effectiveness, a rigorous test of its performance was conducted at the pack level, which allows for the translation of cell-level predictions into real-life problems with hundreds of cells in-series conditions possible. The best models achieve a root mean square error (RMSE) of less than 0.5 test error and approximately 0.1% average percentage error (APE), with maximum absolute errors (MAE) of 2% on the test dataset, accurately estimating SOC under dynamic operating and aging conditions with widely varying operational profiles. These results demonstrate the power of the self-attention Transformer-based model to predict the behavior of complex multiphysics and multiscale battery systems.

show abstract

Transformers in Time-series Analysis: A Tutorial

Cited by 5 publications

References 50 publications

Deep Learning Models for Time-Series Forecasting of RF-EMF in Wireless Networks

Deep Learning Models for Time-Series Forecasting of RF-EMF in Wireless Networks

Cloud-Based Deep Learning for Co-Estimation of Battery State of Charge and State of Health

Spatial-Temporal Self-Attention Transformer Networks for Battery State of Charge Estimation

Contact Info

Product

Resources

About