Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen 2019
DOI: 10.18653/v1/d19-1380
|View full text |Cite
|
Sign up to set email alerts
|

Efficient Sentence Embedding using Discrete Cosine Transform

Abstract: Vector averaging remains one of the most popular sentence embedding methods in spite of its obvious disregard for syntactic structure. While more complex sequential or convolutional networks potentially yield superior classification performance, the improvements in classification accuracy are typically mediocre compared to the simple vector averaging. As an efficient alternative, we propose the use of discrete cosine transform (DCT) to compress word sequences in an order-preserving manner. The lower order DCT … Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
1
1
1
1

Citation Types

0
20
0

Year Published

2020
2020
2021
2021

Publication Types

Select...
6
1
1

Relationship

1
7

Authors

Journals

citations
Cited by 19 publications
(20 citation statements)
references
References 20 publications
0
20
0
Order By: Relevance
“…Rücklé et al (2018) improved the average pooling method by concatenating different power means of word embeddings. Almarwani et al (2019) proposed the use of a Discrete Cosine Transform (DCT) to compress word vectors into sentence embeddings, while retaining word order information.…”
Section: Sentence Embedding Methodsmentioning
confidence: 99%
“…Rücklé et al (2018) improved the average pooling method by concatenating different power means of word embeddings. Almarwani et al (2019) proposed the use of a Discrete Cosine Transform (DCT) to compress word vectors into sentence embeddings, while retaining word order information.…”
Section: Sentence Embedding Methodsmentioning
confidence: 99%
“…Queen (Mikolov et al, 2013). Thus, given word embeddings, cast a sentence as a multidimensional signal over time, in which DCT is used to summarize the general feature patterns in word sequences and compress them into fixed-length vectors (Kayal and Tsatsaronis, 2019;Almarwani et al, 2019).…”
Section: Dct As Sentence Encodermentioning
confidence: 99%
“…However, most of these models, including averaging, disregard structure information, which is an important aspect of any given language. Recently, Almarwani et al (2019) proposed a structure-sensitive sentence encoder, which utilizes Discrete Cosine Transform (DCT) as an efficient alternative to averaging. The authors show that this approach is versatile and scalable because it relies only on word embeddings, which can be easily obtained from large unlabeled data.…”
Section: Introductionmentioning
confidence: 99%
See 1 more Smart Citation
“…DCT is a way to generate document-level representations in an order-preserving manner, adapted from image compression to NLP by Almarwani et al (2019). After mapping an input sequence of real numbers to the coefficients of orthogonal cosine basis functions, low-order coefficients can be used as document embeddings, outperforming vector averaging on most tasks, as shown by the authors.…”
Section: Aggregatorsmentioning
confidence: 99%