Refining Word Representations by Manifold Learning

Chu, Yonghe; Lin, Hongfei; Yang, Liang; Diao, Yuanan; Zhang, Shaowu; Fan, Xiaochao

doi:10.24963/ijcai.2019/749

Cited by 10 publications

(9 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Accordingly, the average random cosine of the left point cloud in Figure 2 approaches 1 while the average random cosine similarity of the right point cloud approaches 0. It is well known that word embedding models have non-zero mean vectors (Yonghe et al, 2019;Liang et al, 2021). In the case of GPT-2 embeddings obtained from the WikiText-2 corpus (Merity et al, 2016), we find values in the mean vector range from −32.36 to 198.19.…”

Section: Word Embeddingsmentioning

confidence: 63%

IsoScore: Measuring the Uniformity of Embedding Space Utilization

Rudman¹,

Gillman²,

Rayne³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

The recent success of distributed word representations has led to an increased interest in analyzing the properties of their spatial distribution. Several studies have suggested that contextualized word embedding models do not isotropically project tokens into vector space. However, current methods designed to measure isotropy, such as average random cosine similarity and the partition score, have not been thoroughly analyzed and are not appropriate for measuring isotropy. We propose IsoScore: a novel tool that quantifies the degree to which a point cloud uniformly utilizes the ambient vector space. Using rigorously designed tests, we demonstrate that IsoScore is the only tool available in the literature that accurately measures how uniformly distributed variance is across dimensions in vector space. Additionally, we use IsoScore to challenge a number of recent conclusions in the NLP literature that have been derived using brittle metrics of isotropy. We caution future studies from using existing tools to measure isotropy in contextualized embedding space as resulting conclusions will be misleading or altogether inaccurate.

show abstract

Section: Word Embeddingsmentioning

confidence: 63%

IsoScore: Measuring the Uniformity of Embedding Space Utilization

Rudman¹,

Gillman²,

Rayne³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

show abstract

“…In addition, some embeddings refinement methods were proposed to generate better semantic word representations. For instance, a word vector refinement model to correct the pre-trained word embedding by using manifold learning was proposed to bring the similarity of words in the Euclidean space closer to word semantics [33]. To learn sentiment word embeddings, a sentiment embedding refinement model based on adjusting the representations of words was used in sentiment analysis to make the word embeddings closer to words which are semantically similar and bear the same polarities and further away from words with opposing polarities [31].…”

Section: Sentiment Embeddingsmentioning

confidence: 99%

“…In sentiment analysis, word embeddings learned by neural models can capture certain types of sentiment features from input texts [21], [22], [23], [24], [25], [26], [27]. Moreover, many embedding refinement methods have been proposed which refine the pre-trained word embeddings based on external knowledge for a better modeling of sentiment information [17], [28], [29], [30], [31], [32], [33]. However, these methods only focus on learning sentiment embeddings, without considering the contextual relations between targets and aspects in the task of TABSA.…”

Section: Introductionmentioning

confidence: 99%

Embedding Refinement Framework for Targeted Aspect-Based Sentiment Analysis

Liang

Yin

Liu

et al. 2023

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

The state-of-the-art approaches to Targeted Aspect-Based Sentiment Analysis (TABSA) are mostly built on deep neural networks with attention mechanisms. One problem is that embeddings of targets and aspects are either pre-trained from large external corpora or randomly initialized. We argue that affective commonsense knowledge and words indicative of sentiment could be used to learn better target and aspect embeddings. We therefore propose an embedding refinement framework called RAEC (Refining Affective Embedding from Context), in which sentiment concepts extracted from affective commonsense knowledge and word relative location information are incorporated to derive context-affective embeddings. Furthermore, a sparse coefficient vector is exploited in refining the embeddings of targets and aspects separately. In this way, embeddings of targets and aspects can capture the highly relevant affective words. Experimental results on two benchmark datasets show that our framework can be easily integrated with existing embedding-based TABSA models and achieves state-of-the-art results compared to models relying on pre-trained word embeddings or built on other embedding refinement methods.

show abstract

“…In contrast, the SBERT-LP analyzes and solve the cosine metric problem of the SBERT sentence space on a manifold. Our work is inspired by the investigation of local geometry in the word space (Hasan and Curry, 2017;Yonghe et al, 2019). These methods solve semantic problems in word space.…”

Section: Related Workmentioning

confidence: 99%

Locality Preserving Sentence Encoding

Min¹,

Chu²,

Yang³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

Self Cite

View full text Add to dashboard Cite

Although researches on word embeddings have made great progress in recent years, many tasks in natural language processing are on the sentence level. Thus, it is essential to learn sentence embeddings. Recently, Sentence BERT (SBERT) is proposed to learn embeddings on the sentence level, and it uses the inner product (or, cosine similarity) to compute semantic similarity between sentences. However, this measurement cannot well describe the semantic structures among sentences. The reason is that sentences may lie on a manifold in the ambient space rather than distribute in a Euclidean space. Thus, cosine similarity cannot approximate distances on the manifold. To tackle the severe problem, we propose a novel sentence embedding method called Sentence BERT with Locality Preserving (SBERT-LP), which discovers the sentence submanifold from a high-dimensional space and yields a compact sentence representation subspace by locally preserving geometric structures of sentences. We compare the SBERT-LP with several existing sentence embedding approaches from three perspectives: sentence similarity, sentence classification, and sentence clustering. Experimental results and case studies demonstrate that our method encodes sentences better in the sense of semantic structures.

show abstract

Refining Word Representations by Manifold Learning

Cited by 10 publications

References 1 publication

IsoScore: Measuring the Uniformity of Embedding Space Utilization

IsoScore: Measuring the Uniformity of Embedding Space Utilization

Embedding Refinement Framework for Targeted Aspect-Based Sentiment Analysis

Locality Preserving Sentence Encoding

Contact Info

Product

Resources

About