Predicting Human Similarity Judgments Using Large Language Models

Abstract

Similarity judgments provide a well-established method for accessing mental representations, with applications in psychology, neuroscience and machine learning. However, collecting similarity judgments can be prohibitively expensive for naturalistic datasets as the number of comparisons grows quadratically in the number of stimuli. One way to tackle this problem is to construct approximation procedures that rely on more accessible proxies for predicting similarity. Here we leverage recent advances in language models and online recruitment, proposing an efficient domain-general procedure for predicting human similarity judgments based on text descriptions. Intuitively, similar stimuli are likely to evoke similar descriptions, allowing us to use description similarity to predict pairwise similarity judgments. Crucially, the number of descriptions required grows only linearly with the number of stimuli, drastically reducing the amount of data required. We test this procedure on six datasets of naturalistic images and show that our models outperform previous approaches based on visual information.

Is my "red" your "red"?: Unsupervised alignment of qualia structures via optimal transport

“…Recent studies have used large language models (LLMs) to investigate the extent to which language contains information about human perception across sensory modalities by examining the accuracy of LLM's inference about human similarity judgments [14,15,8]. These studies have successfully applied LLMs such as GPT-3 to elicit similarity scores between stimuli, which then have their underlying psychological spaces visualized through application of multidimensional scaling (MDS) [8].…”

Alignment with human representations supports robust few-shot learning

“…When an approximate solution is acceptable, Peterson et al (2018) showed that pre-trained computer vision models can be used to approximate human perceptual similarity judgments over images. Marjieh et al (2022a) showed that human perceptual similarity can be more accurately, but still efficiently, approximated from human-produced natural language de- scriptions of the stimuli of interest (for example by using large language models to estimate similarity over pairs of these descriptions). Marjieh et al (2022b) extended this result to a variety of domains (vision, audio, and video) and measured alignment for hundreds of pre-trained models.…”

“…Although several recent papers have proposed ways to measure (Marjieh et al, 2022a;, explain (Muttenthaler et al, 2022;Kumar et al, 2022), and even improve (Peterson et al, 2018;Fel et al, 2022) the representational alignment of models, few have focused on studying the downstream impact of a model being representationally aligned with humans, and many studies simply rely on the intuition that better alignment leads to better performance to justify pursuing increased alignment. While there is recent evidence to suggest that alignment may help humans learn across domains and perform zero-shot generalization (Aho et al, 2022), there is also evidence to suggest that alignment may not always be beneficial for models, with models scoring low on alignment metrics achieving higher performance on downstream tasks like image classification (Kumar et al, 2022;Muttenthaler et al, 2022;Fel et al, 2022).…”

“…Few-shot transfer learning performance was evaluated on the CIFAR100 (Krizhevsky et al, 2009) test set with n ∈ {1, 5, 10, 20, 40, 80} examples per class used for few-shot learning and the remaining examples used for evaluation. We measure representational alignment of models by computing the three metrics described below on the six image datasets from Peterson et al (2018) and their respective sets of similarity judgments consisting of over 425,000 judgments across 1200 participants, as described by Marjieh et al (2022a). We average each of the three alignment metrics over the six datasets.…”

“…We define representational alignment as the degree to which the latent representations of a model match the latent representations of humans for the same set of stimuli, and refer to models that are representationally aligned with humans as being "human-aligned." Several recent papers have proposed ways to measure (Marjieh et al, 2022a;, explain (Muttenthaler et al, 2022;Kumar et al, 2022), and even improve (Peterson et al, 2018;Fel et al, 2022) the representational alignment of models. However, many models that score low on these alignment metrics still have high performance on downstream tasks like image classification (Kumar et al, 2022;Muttenthaler et al, 2022;Fel et al, 2022).…”