2022
DOI: 10.48550/arxiv.2203.01178
|View full text |Cite
Preprint
|
Sign up to set email alerts
|

DCT-Former: Efficient Self-Attention with Discrete Cosine Transform

Abstract: Since their introduction the Trasformer architectures emerged as the dominating architectures for both natural language processing and, more recently, computer vision applications. An intrinsic limitation of this family of "fully-attentive" architectures arises from the computation of the dot-product attention, which grows both in memory consumption and number of operations as O(n 2 ) where n stands for the input sequence length, thus limiting the applications that require modeling very long sequences. Several… Show more

Help me understand this report
View published versions

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

0
0
0

Publication Types

Select...

Relationship

0
0

Authors

Journals

citations
Cited by 0 publications
references
References 32 publications
0
0
0
Order By: Relevance

No citations

Set email alert for when this publication receives citations?