Similarity judgment within and across categories: A comprehensive model comparison

Abstract

Similarity is one of the most important relations humans perceive, arguably subserving category learning and categorization, generalization and discrimination, judgment and decision making, and other cognitive functions. Researchers have proposed a wide range of representations and processes that could be at play in similarity judgment, yet have not comprehensively compared the power of these representations and processes for predicting similarity within and across different semantic categories. We performed such a comparison by pairing eight prominent vector semantic representations with seven established similarity metrics that could operate on these representations, as well as supervised methods for dimensional weighting in the similarity function. This approach yields a factorial model structure with 56 distinct representation-process pairs, which we tested on a novel dataset of similarity judgments between pairs of co-hyponymic words in eight categories. We found that cosine similarity and Pearson correlation were the overall best performing unweighted similarity functions, and that word vectors derived from free association norms often outperformed word vectors derived from text (including those specialized for similarity). Importantly, models that used human similarity judgments to learn category-specific weights on dimensions yielded substantially better predictions than all unweighted approaches across all types of similarity functions and representations, although dimension weights did not generalize well across semantic categories, suggesting strong category context effects in similarity judgment. We discuss implications of these results for cognitive modeling and natural language processing, as well as for theories of representations and processes involved in similarity.

Transformer Networks of Human Conceptual Knowledge

“…Our final application showed that BERT was able to predict patterns in similarity judgment that are problematic for existing distributed semantics models. These patterns involve asymmetry in similarity judgment (Whitten et al, 1979), the distinction between association and similarity (Hill et al, 2015), and the measurement of similarity within (rather than across) categories (Richie & Bhatia, 2020).…”

“…Word vector representations learned through these methods have been enormously successful in psychological applications (for reviews see Bhatia et al, 2019;Günther et al, 2019;Jones et al, 2015;Lenci, 2018;Mandera et al, 2017). For example, cosine similarity between word vectors correlates with Likert-scale judgments of words' similarity and relatedness (Richie & Bhatia, 2020;Hill et al, 2015), strength of semantic priming in, e.g., the lexical decision task, as measured by reaction times (Jones et al, 2006;Mandera et al, 2017), and even with probability of recall given a cue in free association, or given an earlier recalled item in list and category recall (although there are often better ways to use word vectors for such tasks rather than simply computing cosine; see Nematzadeh et al, 2017 andJones et al, 2018). Semantic judgments about words (e.g.…”

“…To address the limitations of existing research, we recently examined participant similarity judgments for word pairs with a shared category (Richie & Bhatia, 2020). Specifically, we offered participants 2,040 word pairs, each of which involved concepts from one of seven categories: furniture, clothing, birds, vegetables, sports, vehicles, and fruit.…”

“…In the current paper we wanted to test whether BERT was able to avoid these issues. For this purpose, we applied it to the word pairs in Richie and Bhatia (2020) is a [WORD 1] sentences, and correlated these predictions separately for each category. These predictions are shown in Figure 9C.…”

Can we detect conditioned variation in political speech? two kinds of discussion and types of conversation

“…One popular DS model is word2vec, which projects each word of its input into a common, lower-dimensional space [61,62]. Words that are closer together in this space are more semantically similar [63,64]. One common way to measure this distance is using cosine similarity, or the cosine of the angle between the vector representations of two words.…”