P-SIF: Document Embeddings Using Partition Averaging

Gupta, Vivek; Saw, Ankit; Nokhiz, Pegah; Netrapalli, Praneeth; Rai, Piyush; Talukdar, Partha

doi:10.1609/aaai.v34i05.6292

Cited by 18 publications

(13 citation statements)

References 18 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our work, we adhere to the linear evaluation protocol and use a simple logistic regression classifier in evaluating word representations (Arora et al, 2019;Gupta et al, 2020), whereas (Mu and Viswanath, 2018) use a neural network as their classifier. The red colored cells in Table 5 denote the cases where the original embeddings outperformed their Post Processed (PPA) counterparts.…”

Section: Sentence Classification Tasksmentioning

confidence: 99%

On Dimensional Linguistic Properties of the Word Embedding Space

Raunak

Kumar

Gupta

et al. 2020

Proceedings of the 5th Workshop on Representation Learning for NLP

Self Cite

View full text Add to dashboard Cite

Word embeddings have become a staple of several natural language processing tasks, yet much remains to be understood about their properties. In this work, we analyze word embeddings in terms of their principal components and arrive at a number of novel and counterintuitive observations. In particular, we characterize the utility of variance explained by the principal components as a proxy for downstream performance. Furthermore, through syntactic probing of the principal embedding space, we show that the syntactic information captured by a principal component does not correlate with the amount of variance it explains. Consequently, we investigate the limitations of variance based embedding post-processing, used in a few algorithms such as (Mu and Viswanath, 2018;Raunak et al., 2019) and demonstrate that such postprocessing is counter-productive in sentence classification and machine translation tasks. Finally, we offer a few precautionary guidelines on applying variance based embedding post-processing and explain why non-isotropic geometry might be integral to word embedding performance.

show abstract

Section: Sentence Classification Tasksmentioning

confidence: 99%

On Dimensional Linguistic Properties of the Word Embedding Space

Raunak

Kumar

Gupta

et al. 2020

Proceedings of the 5th Workshop on Representation Learning for NLP

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although we focused on re-evaluating WMD, our findings are beneficial for other topics as well. First, the datasets released by the WMD paper have been used in non-OT studies as well [72,50,31]. Our findings regarding the duplicate documents are important for proper evaluations of other methods as well.…”

Section: Discussionmentioning

confidence: 99%

Re-evaluating Word Mover's Distance

Sato¹,

Yamada²,

Kashima

2021

Preprint

View full text Add to dashboard Cite

The word mover's distance (WMD) is a fundamental technique for measuring the similarity of two documents. As the crux of WMD, it can take advantage of the underlying geometry of the word space by employing an optimal transport formulation. The original study on WMD reported that WMD outperforms classical baselines such as bag-of-words (BOW) and TF-IDF by significant margins in various datasets. In this paper, we point out that the evaluation in the original study could be misleading. We re-evaluate the performances of WMD and the classical baselines and find that the classical baselines are competitive with WMD if we employ an appropriate preprocessing, i.e., L1 normalization. However, this result is not intuitive. WMD should be superior to BOW because WMD can take the underlying geometry into account, whereas BOW cannot. Our analysis shows that this is due to the high-dimensional nature of the underlying metric. We find that WMD in high-dimensional spaces behaves more similarly to BOW than in low-dimensional spaces due to the curse of dimensionality.Preprint. Under review.

show abstract

“…First, we use word weights in constructing semantically similar groups and can incorporate different weighting scheme in our model, such as SIF [11], GEM [12]. Second, different clustering schemes such as the Gaussian mixture model and dictionary learning can be utilized to construct semantically similar groups [13], [31]. Finally, the intra-group descriptor can be replaced by methods like VLAWE [18] and p-means [14].…”

Section: Discussionmentioning

confidence: 99%

Efficient Sentence Embedding via Semantic Subspace Analysis

Wang

Chen

Wang

et al. 2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

A novel sentence embedding method built upon semantic subspace analysis, called semantic subspace sentence embedding (S3E), is proposed in this work. Given the fact that word embeddings can capture semantic relationship while semantically similar words tend to form semantic groups in a high-dimensional embedding space, we develop a sentence representation scheme by analyzing semantic subspaces of its constituent words. Specifically, we construct a sentence model from two aspects. First, we represent words that lie in the same semantic group using the intra-group descriptor. Second, we characterize the interaction between multiple semantic groups with the inter-group descriptor. The proposed S3E method is evaluated on both textual similarity tasks and supervised tasks. Experimental results show that it offers comparable or better performance than the state-of-the-art. The complexity of our S3E method is also much lower than other parameterized models.

show abstract

P-SIF: Document Embeddings Using Partition Averaging

Cited by 18 publications

References 18 publications

On Dimensional Linguistic Properties of the Word Embedding Space

On Dimensional Linguistic Properties of the Word Embedding Space

Re-evaluating Word Mover's Distance

Efficient Sentence Embedding via Semantic Subspace Analysis

Contact Info

Product

Resources

About