Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Arora, Sanjeev; Li, Yuanzhi; Liang, Yingyu; Ma, Tengyu; Risteski, Andrej

doi:10.48550/arxiv.1601.03764

Cited by 14 publications

(26 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sparse coding is usually solved for a given K and k by using alternating minimization such as k-svd (Aharon et al 2006) to find the A i s that minimize the following L 2 -reconstruction error: vw − K j=1 α (w,j) Aj . (Arora et al 2016b) show that multiple senses of a word reside as a linear superposition within the word embedding and can be recovered by simple sparse coding. Therefore, one can use the sparse coding of word vectors to detect multiple senses of words.…”

Section: Averaging Vs Partition Averagingmentioning

confidence: 99%

“…10 We use script for preprocessing the dataset. 11 We consider several embedding baselines mostly taken from (Mekala et al 2017;Wu et al 2018;Arora et al 2016b). More details on experimental settings and hyper-parameters' values are described in the Supplementary material 5 .…”

Section: Text Classification Taskmentioning

confidence: 99%

“…2. Total number of topics in the entire corpus is K, which can be determined by sparse dictionary learning as shown by (Arora et al 2016b) over word vectors vw. 3.…”

Section: Derivation Of P-sif Embeddingsmentioning

confidence: 99%

“…However, the above assumptions do not hold true for a document with multiple sentences where one can expect to have more frequent jumps during a random walk due to topic change. 16 Instead of assuming a single context for the whole document c h , we assume that the total number of contexts over a given corpus is bounded by the number of topics K (as shown by (Arora et al 2016b)), and the random walk can perform jumps to switch context from one context to the rest of K − 1 contexts. The partition function remains the same in all directions only for the words emerging from the same context cj instead of the words coming from all the K contexts.…”

Section: Derivation Of P-sif Embeddingsmentioning

confidence: 99%

See 3 more Smart Citations

P-SIF: Document Embeddings Using Partition Averaging

Gupta¹,

Saw²,

Nokhiz³

et al. 2020

Preprint

View full text Add to dashboard Cite

Simple weighted averaging of word vectors often yields effective representations for sentences which outperform sophisticated seq2seq neural models in many tasks. While it is desirable to use the same method to represent documents as well, unfortunately, the effectiveness is lost when representing long documents involving multiple sentences. One of the key reasons is that a longer document is likely to contain words from many different topics; hence, creating a single vector while ignoring all the topical structure is unlikely to yield an effective document representation. This problem is less acute in single sentences and other short text fragments where the presence of a single topic is most likely. To alleviate this problem, we present P-SIF, a partitioned word averaging model to represent long documents. P-SIF retains the simplicity of simple weighted word averaging while taking a document's topical structure into account. In particular, P-SIF learns topic-specific vectors from a document and finally concatenates them all to represent the overall document. We provide theoretical justifications on the correctness of P-SIF. Through a comprehensive set of experiments, we demonstrate P-SIF's effectiveness compared to simple weighted averaging and many other baselines.

show abstract

Section: Averaging Vs Partition Averagingmentioning

confidence: 99%

Section: Text Classification Taskmentioning

confidence: 99%

“…2. Total number of topics in the entire corpus is K, which can be determined by sparse dictionary learning as shown by (Arora et al 2016b) over word vectors vw. 3.…”

Section: Derivation Of P-sif Embeddingsmentioning

confidence: 99%

Section: Derivation Of P-sif Embeddingsmentioning

confidence: 99%

See 2 more Smart Citations

P-SIF: Document Embeddings Using Partition Averaging

Gupta¹,

Saw²,

Nokhiz³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Given a target word, the task (Arora et al, 2016;Sun et al, 2017) is to identify the true context corresponding to a sense of the target word out of 10 other randomly selected false contexts, where a context is presented by similar words. For example, two of the true contexts for the target word bank are water,land,river,... and institution,deposits,money.... We use the R1 dataset from Sun et al (2017), which consists of 137 word types and 535 queries.…”

Section: Word Context Relevance (Wcr)mentioning

confidence: 99%

Efficient Graph-based Word Sense Induction by Distributional Inclusion Vector Embeddings

Chang

Agrawal

Ganesh

et al. 2018

Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12)

View full text Add to dashboard Cite

Word sense induction (WSI), which addresses polysemy by unsupervised discovery of multiple word senses, resolves ambiguities for downstream NLP tasks and also makes word representations more interpretable. This paper proposes an accurate and efficient graphbased method for WSI that builds a global non-negative vector embedding basis (which are interpretable like topics) and clusters the basis indexes in the ego network of each polysemous word. By adopting distributional inclusion vector embeddings as our basis formation model, we avoid the expensive step of nearest neighbor search that plagues other graph-based methods without sacrificing the quality of sense clusters. Experiments on three datasets show that our proposed method produces similar or better sense clusters and embeddings compared with previous state-of-theart methods while being significantly more efficient.

show abstract

Does Protest Influence Political Speech? Evidence from UK Climate Protest, 2017–2019

Barrie,

Fleming,

Rowan

2023

Brit. J. Polit. Sci.

View full text Add to dashboard Cite

How does protest affect political speech? Protest is an important form of political claim-making, yet our understanding of its influence on how individual legislators communicate remains limited. Our paper thus extends a theoretical framework on protests as information about voter preferences, and evaluates it using crowd-sourced protest data from the 2017–2019 Fridays for Future protests in the UK. We combine these data with ~2.4m tweets from 553 legislators over this period and text data from ~150k parliamentary speech records. We find that local protests prompted MPs to speak more about the climate, but only online. These results demonstrate that protest can shape the timing and substance of political communication by individual elected representatives. They also highlight an important difference between legislators' offline and online speech, suggesting that more work is needed to understand how political strategies differ across these arenas.

show abstract

Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Cited by 14 publications

References 0 publications

P-SIF: Document Embeddings Using Partition Averaging

P-SIF: Document Embeddings Using Partition Averaging

Efficient Graph-based Word Sense Induction by Distributional Inclusion Vector Embeddings

Does Protest Influence Political Speech? Evidence from UK Climate Protest, 2017–2019

Contact Info

Product

Resources

About