Word2Sense: Sparse Interpretable Word Embeddings

Panigrahi, Abhishek; Simhadri, Harsha Vardhan; Bhattacharyya, Chiranjib

doi:10.18653/v1/p19-1570

Cited by 40 publications

(32 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subramanian et al [37] utilize denoising k-sparse autoencoder to generate efficient and interpretable distributed word representations. The work by Panigrahi et al [26] is to the best our knowledge, among the existing research, closest to our work. The authors propose Word2Sense word embeddings in which each dimension of the embedding space corresponds to a fine-grained sense, and the non-negative value of the embedding along a dimension represents the relevance of the sense to the word.…”

Section: Interpretable Word Embeddingsmentioning

confidence: 52%

The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings

Mathew

Sikdar

Lemmerich

et al. 2020

Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

We introduce 'POLAR' -a framework that adds interpretability to pre-trained word embeddings via the adoption of semantic differentials. Semantic differentials are a psychometric construct for measuring the semantics of a word by analysing its position on a scale between two polar opposites (e.g., cold -hot, soft -hard). The core idea of our approach is to transform existing, pre-trained word embeddings via semantic differentials to a new "polar" space with interpretable dimensions defined by such polar opposites. Our framework also allows for selecting the most discriminative dimensions from a set of polar dimensions provided by an oracle, i.e., an external source. We demonstrate the effectiveness of our framework by deploying it to various downstream tasks, in which our interpretable word embeddings achieve a performance that is comparable to the original word embeddings. We also show that the interpretable dimensions selected by our framework align with human judgement. Together, these results demonstrate that interpretability can be added to word embeddings without compromising performance. Our work is relevant for researchers and engineers interested in interpreting pre-trained word embeddings. CCS CONCEPTS• Computing methodologies → Machine learning approaches. KEYWORDS word embeddings, neural networks, interpretable, semantic differential ACM Reference Format:

show abstract

Section: Interpretable Word Embeddingsmentioning

confidence: 52%

The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings

Mathew

Sikdar

Lemmerich

et al. 2020

Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

show abstract

“…Research has also been done to produce more interpretable static word embeddings e.g. (Ş enel et al, 2020;Panigrahi et al, 2019). For contextual embeddings, Aloui et al (2020) produced embeddings with semantic super-senses as dimensions, but these are quite broad.…”

Section: Interpretable Word Embeddingsmentioning

confidence: 99%

“…Whilst efforts have been made to produce more interpretable embeddings e.g. (Ş enel et al, 2020;Panigrahi et al, 2019), the general approach has been to interpret them in relation to each-other. For example, the relative distance between word embeddings can indicate their semantic similarity (Schnabel et al, 2015).…”

Section: Introductionmentioning

confidence: 99%

Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings

Turton¹,

Smith²,

Vinson³

2021

Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)

View full text Add to dashboard Cite

Models based on the transformer architecture, such as BERT, have marked a crucial step forward in the field of Natural Language Processing. Importantly, they allow the creation of word embeddings that capture important semantic information about words in context. However, as single entities, these embeddings are difficult to interpret and the models used to create them have been described as opaque. Binder and colleagues proposed an intuitive embedding space where each dimension is based on one of 65 core semantic features. Unfortunately, the space only exists for a small data-set of 535 words, limiting its uses. Previous work (Utsumi, 2018(Utsumi, , 2020Turton et al., 2020) has shown that Binder features can be derived from static embeddings and successfully extrapolated to a large new vocabulary. Taking the next step, this paper demonstrates that Binder features can be derived from the BERT embedding space. This provides two things; (1) semantic feature values derived from contextualised word embeddings and (2) insights into how semantic features are represented across the different layers of the BERT model.

show abstract

“…It is noteworthy that we constrain the nonnegativity of B t and C t to learn sparse interpretable word embeddings (Murphy et al, 2012;Luo et al, 2015), so as to capture the polysemous nature of words (Panigrahi et al, 2019). With non-negativity constraints, words are represented by limited dimensions (Murphy et al, 2012).…”

Section: Objective Functionmentioning

confidence: 99%

Lifelong Learning of Topics and Domain-Specific Word Embeddings

Qin¹,

Lu²,

Rao³

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Lifelong topic models mainly focus on indomain text streams in which each chunk only contains documents from a single domain. To overcome data diversity of the in-domain corpus, most of the existing methods exploit the information from limited sources in a separate and heuristic manner. In this study, we develop a lifelong collaborative model (LCM) based on non-negative matrix factorization to accurately learn topics and domain-specific word embeddings. LCM particularly investigates:(1) developing a knowledge graph based on the semantic relationships among words in the lifelong learning process, so as to accumulate global context information discovered by topic models and local context information reflected by context word embeddings from previous domains, and (2) developing a subword graph based on byte pair encoding and pairwise word relationships to exploit subword information of words in the current in-domain corpus. To the best of our knowledge, we are the first to collaboratively learn topics and word embeddings via lifelong learning. Experiments on real-world in-domain text streams validate the effectiveness of our method.

show abstract

Word2Sense: Sparse Interpretable Word Embeddings

Cited by 40 publications

References 33 publications

The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings

The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings

Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings

Lifelong Learning of Topics and Domain-Specific Word Embeddings

Contact Info

Product

Resources

About