What Analogies Reveal about Word Vectors and their Compositionality

Abstract

Analogy completion via vector arithmetic has become a common means of demonstrating the compositionality of word embeddings. Previous work have shown that this strategy works more reliably for certain types of analogical word relationships than for others, but these studies have not offered a convincing account for why this is the case. We arrive at such an account through an experiment that targets a wide variety of analogy questions and defines a baseline condition to more accurately measure the efficacy of our system. We find that the most reliably solvable analogy categories involve either 1) the application of a morpheme with clear syntactic effects, 2) male-female alternations, or 3) named entities. These broader types do not pattern cleanly along a syntacticsemantic divide. We suggest instead that their commonality is distributional, in that the difference between the distributions of two words in any given pair encompasses a relatively small number of word types. Our study offers a needed explanation for why analogy tests succeed and fail where they do and provides nuanced insight into the relationship between word distributions and the theoretical linguistic domains of syntax and semantics.

Bringing Order to Neural Word Embeddings with Embeddings Augmented by Random Permutations (

“…country:capital, male:female). We obtained these sets from the distribution described in Finley et al (2017) 6 , in which only the first correct answer to questions with multiple correct answers in BATS is retained, and used a parallelized implementation of the widely used vector offset method, in which for a given proportional analogy a:b:c:d, all word vectors in the space are rank-ordered in accordance with their cosine similarity to the vector − −−−−− → c + b − a. We report average accuracy, where a result is considered accurate if d is the top-ranked result aside from a, b and c. 7 To evaluate the effects of encoding word order on the relative distance between terms, we used a series of widely used reference sets that mediate comparison between human and machine estimates of pairwise similarity and relatedness between term pairs.…”

Artificial Intelligence in mental health and the biases of language based models

“…Word embeddings provide a mathematical model for inferring semantic meaning from words [46][47][48][49][50]. The word embeddings are a key component of the NLP model and they rely on the distributional hypothesis which states "a word is characterized by the company it keeps".…”

Semantic Source Code Models Using Identifier Embeddings

“…Text Normalization: Lemmatization and stemming are standard normalization techniques employed in order to mitigate the noise produced by grammatical inflections in a variety of natural language processing tasks. However, in continuous vector space models, this type of normalization could lead to information loss as inflections may capture relational analogies, e.g., nominal plural analogies, such as "dog is to dogs what horse is to horses" [40]. Inflection phenomena are not equally pervasive in programming scripts; still source code identifiers do incorporate aspects of inflection, e.g., a class named "Node" versus a collection which is named "nodes" and holds instances of "Node" objects.…”