Multivariate time series data imputation using attention-based mechanism

Zhao, Jian‐xin; Rong, Chuitian; Lin, Chunbin; Dang, Xin

doi:10.1016/j.neucom.2023.126238

Cited by 9 publications

(1 citation statement)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After that, the normalized data goes through multi-head self-attention process, where the mechanism of the self-attention is further described in ref. [18,27]. The result of multi-head attention is then added to the data prior to the self-attention process, and the sum of both values is subjected to normalization again.…”

Section: Model Construction and Trainingmentioning

confidence: 99%

Modelling and Next-Value Prediction of Beam Propagation from Grating Structures using Simplified Transformer Model

Lim,

Tan

2024

Preprint

View full text Add to dashboard Cite

In this study, simplified transformer model is used to perform next-value prediction on light coupled out from silicon photonics grating to free space. Finite-difference time-domain (FDTD) simulation is performed to simulate the electric field (E-field) in laser light coupled from gratings with pitches of 0.6, 0.8, 1.0, 1.2, 1.4 and 1.6 µm, to free-space. Only E-field distribution from 0.6 µm is used in model training, and the trained transformer model is used to predict the E-field from the rest of the gratings. Prediction of accuracy up to 81.7% is obtained. The time taken for model training is 1908.4 seconds, which is significantly shorter than the conventional three-dimensional FDTD simulation that takes up to several hours. If the prediction accuracy can be compromised, the transformer model can be trained with stepped datasets, where the training time can be reduced to 56 and 114 seconds. We demonstrated that the transformer model can be used to perform next-value E-field prediction using minimal training data. The developed and trained transformer model can be integrated to the state-of-the-art FDTD software to further expedite the existing FDTD simulation.

show abstract

Section: Model Construction and Trainingmentioning

confidence: 99%