A Simple and Effective Positional Encoding for Transformers

Chen, Pu-Chin; Tsai, Henry; Bhojanapalli, Srinadh; Chung, Hyung Won; Chang, Yuan-Hao; Ferng, Chun-Sung

doi:10.48550/arxiv.2104.08698

Cited by 3 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the original version of a transformer, this projection is replaced by pre-trained token embeddings 37,41 . Afterwards, an encoding layer 90 added positional information that would have been lost in the attention module 37 . After the positional encoding, a multi-head attention module calculated attention weights encoding temporal dynamics.…”

Section: Model Architecturesmentioning

confidence: 99%

TRAPOD: A Transformer Architecture Exploits Intraoperative Temporal Dynamics Improving the Prediction of Postoperative Delirium

Giesa,

Sekutowicz,

Rubarth

et al. 2024

Preprint

View full text Add to dashboard Cite

Motivation. Patients who experienced postoperative delirium (POD) are at higher risk of poor outcomes like dementia or death. Previous machine learning (ML) models predicting POD mostly relied on time-aggregated features. Objective. We assessed the potential of temporal patterns in clinical parameters during surgeries to predict POD. Methods. Long short-term memory (LSTM) and transformer models, directly consuming time series, were compared to multi-layer perceptrons (MLPs) trained on time-aggregated features. We also fitted hybrid models, fusing either LSTM or transformer models with MLPs. Univariate Spearman’s rank correlations and linear mixed-effect models establish the importance of individual features that we compared to transformers’ attention weights. Results. Best performance was achieved by a transformer architecture ingesting 30 minutes of intraoperative parameter sequences. Systolic invasive blood pressure and given opioids marked the most important input variables, in line with univariate feature importances. Conclusion. Intraoperative temporal dynamics of clinical parameters, exploited by a tailored transformer architecture named TRAPOD, are critical for the accurate prediction of POD.

show abstract

Section: Model Architecturesmentioning

confidence: 99%

TRAPOD: A Transformer Architecture Exploits Intraoperative Temporal Dynamics Improving the Prediction of Postoperative Delirium

Giesa,

Sekutowicz,

Rubarth

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…[53] Because each word matches sine and cosine curves of different periods using the transformation equation of the trigonometric function, different positions obtain unique positional encoding. In addition, the latest research reports on advanced positional encoding, such as Decoupled posItional attEntion for Transformers (DIET) [54] and Position Encoding Generator (PEG). [55]…”

Section: Positional Encodingmentioning

confidence: 99%

Understanding the brain with attention: A survey of transformers in brain sciences

Chen,

Wang,

Chen

et al. 2023

Brain-X

View full text Add to dashboard Cite

Owing to their superior capabilities and advanced achievements, Transformers have gradually attracted attention with regard to understanding complex brain processing mechanisms. This study aims to comprehensively review and discuss the applications of Transformers in brain sciences. First, we present a brief introduction of the critical architecture of Transformers. Then, we overview and analyze their most relevant applications in brain sciences, including brain disease diagnosis, brain age prediction, brain anomaly detection, semantic segmentation, multi‐modal registration, functional Magnetic Resonance Imaging (fMRI) modeling, Electroencephalogram (EEG) processing, and multi‐task collaboration. We organize the model details and open sources for reference and replication. In addition, we discuss the quantitative assessments, model complexity, and optimization of Transformers, which are topics of great concern in the field. Finally, we explore possible future challenges and opportunities, exploiting some concrete and recent cases to provoke discussion and innovation. We hope that this review will stimulate interest in further research on Transformers in the context of brain sciences.

show abstract

“…It might be difficult to effectively capture the global context or long-range dependencies, which are essential for comprehending complex scenes or capturing relationships between far-off objects. To address some of these issues, involution neural networks (INNs), which are computationally efficient and more parallelizable, are used as alternatives to CNNs (Chen et al, 2021). Unlike standard convolution kernels, which are spatially agnostic and channel specific, involution kernels are more suitable for capturing long-range spatial information while minimizing network parameters.…”

Section: Introductionmentioning

confidence: 99%

Plant disease detection using leaf images and an involutional neural network

Pradhan,

Kumar,

Kumar

et al. 2024

ECJ

View full text Add to dashboard Cite

The human population and domestic animals rely heavily on agriculture for their food and livelihood. Agriculture is an important contributor to the national economy of many countries. Plant diseases lead to a significant reduction in agricultural yield, posing a threat to global food security. It is crucial to detect plant diseases in a timely manner to prevent economic losses. Expert diagnosis and pathogen analysis are widely used for the detection of diseases in plants. However, both expert diagnosis and pathogen analysis rely on the real-time investigation experience of experts, which is prone to errors. In this work, an image analysis-based method is proposed for detecting and classifying plant diseases using an involution neural network and self-attention-based model. This method uses digital images of plant leaves and identifies diseases on the basis of image features. Different diseases affect leaf characteristics in different ways; therefore, their visual patterns are highly useful in disease recognition. For rigorous evaluation of the method, leaf images of different crops, including apple, grape, peach, cherry, corn, pepper, potato, and strawberry, are taken from a publicly available PlantVillage dataset to train the developed model. The experiments are not performed separately for different crops; instead, the model is trained to work for multiple crops. The experimental results demonstrate that the proposed method performed well, with an average classification accuracy of approximately 98.73% (κ = 98.04) for 8 different crops with 23 classes. The results are also compared with those of several existing methods, and it is found that the proposed method outperforms the other methods considered in this work.

show abstract

A Simple and Effective Positional Encoding for Transformers

Cited by 3 publications

References 9 publications

TRAPOD: A Transformer Architecture Exploits Intraoperative Temporal Dynamics Improving the Prediction of Postoperative Delirium

TRAPOD: A Transformer Architecture Exploits Intraoperative Temporal Dynamics Improving the Prediction of Postoperative Delirium

Understanding the brain with attention: A survey of transformers in brain sciences

Plant disease detection using leaf images and an involutional neural network

Contact Info

Product

Resources

About