Syntax-Aware Multi-Sense Word Embeddings for Deep Compositional Models of Meaning

Abstract

Deep compositional models of meaning acting on distributional representations of words in order to produce vectors of larger text constituents are evolving to a popular area of NLP research. We detail a compositional distributional framework based on a rich form of word embeddings that aims at facilitating the interactions between words in the context of a sentence. Embeddings and composition layers are jointly learned against a generic objective that enhances the vectors with syntactic information from the surrounding context. Furthermore, each word is associated with a number of senses, the most plausible of which is selected dynamically during the composition process. We evaluate the produced vectors qualitatively and quantitatively with positive results. At the sentence level, the effectiveness of the framework is demonstrated on the MSRPar task, for which we report results within the state-of-the-art range.

Semantically-informed distance and similarity measures for paraphrase plagiarism identification

“…Additionally, this table shows the state-of-the-art results for the two used datasets. As noticed, the results from our combined approach are close to the reference results, nonetheless, ours is a much more simple approach (for example, [7] reports a recursive neural network using syntax-aware and multi-sense word embeddings). Table 5: F 1 results in several paraphrase categories using different similarity and distance measures.…”

Syntax Encoding with Application in Authorship Attribution

“…Furthermore, the recent distributed word embedding techniques, such as Glove (Pennington et al, 2014) and W2V (Mikolov et al, 2013), have been shown to encode limited syntax knowledge of the given corpus (Andreas and Klein, 2014). This shortcoming has also promoted recent research on creating syntax-aware word embedding, which enhances the distributed embedding vectors with position information of the word within its surrounding context (Cheng and Kartsaklis, 2015), which again encodes limited syntax information.…”

Implicit Subjective and Sentimental Usages in Multi-sense Word Embeddings

“…Multi-sense word embedding is also a popular way to represent polysemous words (Reisinger and Mooney, 2010;Huang et al, 2012;Neelakantan et al, 2014;Guo et al, 2014;Li and Jurafsky, 2015;Iacobacci et al, 2015;Cheng and Kartsaklis, 2015;Lee and Chen, 2017). However, these method using contextual difference for sense clustering to decide senses are so sensitive to contextual variation and usage of word, therefore may embed a single sense into several vectors.…”

“…Multi-sense word embeddings are popular choices to represent polysemous words (Reisinger and Mooney, 2010;Huang et al, 2012;Neelakantan et al, 2014;Cheng and Kartsaklis, 2015;Lee and Chen, 2017). These methods learn senses of words automatically by clustering contexts they appear in.…”