ViSpa (Vision Spaces): A computer-vision-based representation system for individual images and concept prototypes, with large-scale evaluation.

Abstract

Quantitative, data-driven models for mental representations have long enjoyed popularity and success in psychology (e.g., distributional semantic models in the language domain), but have largely been missing for the visual domain. To overcome this, we present ViSpa (Vision Spaces), high-dimensional vector spaces that include vision-based representation for naturalistic images as well as concept prototypes. These vectors are derived directly from visual stimuli through a deep convolutional neural network trained to classify images and allow us to compute vision-based similarity scores between any pair of images and/or concept prototypes. We successfully evaluate these similarities against human behavioral data in a series of large-scale studies, including off-line judgments-visual similarity judgments for the referents of word pairs (Study 1) and for image pairs (Study 2), and typicality judgments for images given a label (Study 3)-as well as online processing times and error rates in a discrimination (Study 4) and priming task (Study 5) with naturalistic image material. ViSpa similarities predict behavioral data across all tasks, which renders ViSpa a theoretically appealing model for vision-based representations and a valuable research tool for data analysis and the construction of experimental material: ViSpa allows for precise control over experimental material consisting of images and/or words denoting imageable concepts and introduces a specifically vision-based similarity for word pairs. To make ViSpa available to a wide audience, this article (a) includes (video) tutorials on how to use ViSpa in R and (b) presents a user-friendly web interface at http://vispa.fritzguenther.de.

The Flickr frequency norms: What 17 years of images tagged online tell us about lexical processing

“…Having such a source of data available is highly desirable for psychological studies, as it allows to bypass the loophole of predicting behavioural data (e.g., lexical processing time) from other behavioural data (ratings) (Jones et al, 2015;Westbury, 2016) -which would leave us staying essentially at the same epistemological level of description rather than providing explanations rooted at a more basic level. Instead word frequency constitutes an independent and, more importantly, primary source of data that is supposed to act as the foundation for ratings as a secondary variable (i.e., a function of this input) (see Günther et al, 2022). On a practical level, the most crucial benefit of this methodology is that it can be, in principle, applied to extract estimates of Flickr frequency for any existing word in any language.…”

“…Chen & Gupta, 2015;Das & Clark, 2018). Furthermore, Flickr images constitute a quite large part of ImageNet (Deng et al, 2009), a large-scale database of labelled images adopting the hierarchical category structure of WordNet (Miller, 1998) and designed for use in visual object recognition research (e.g., Krizhevsky et al, 2012), but also adopted to build up prototypical vision-based representations for concepts to be used in psychological research (Anderson et al, 2015;Günther et al, 2022;Petilli et al, 2021).…”

“…Petilli, M. A., Günther, F., & Marelli, M. (2022). The Flickr frequency norms: What 17 years of images tagged online tell us about lexical processing.…”

Word-association norms for 1,100 French words with varying levels of concreteness

“…Psycholinguistic experiments which study the organisation of concepts in the mental lexicon often rely on normative stimuli. Although there are numerous types of relationships between concepts from lexical and distributional to sensorimotor components (Günther et al, 2022;Landauer & Dumais, 1997;Lund & Burgess, 1996;Mikolov, Chen, et al, 2013;Villani et al, 2021;Wingfield & Connell, 2022a, 2022b, two types of norms have a long tradition in the field: semantic similarity and word-association norms. Semantic similarity refers to the overlap between the defining features of concepts (e.g.,<bee> and <wasp> are semantically similar).…”

Visual experience modulates the sensitivity to the distributional history of words in natural language

“…While the amodal and embodied accounts have been traditionally contrasted, recent views acknowledge that both sensorimortor and linguistic experiences would be crucial in the development of semantic knowledge. However, the relative contribution of each type of experiential trace is still debated (Andrews, 2009;Davis & Yee, 2020;Günther et al, 2022;Günther et al, 2023;. This is especially due to the fact that humans' reliance and sensitivity to perceptual or linguistic experience may flexibly vary depending on situational and contextual demands (Kemmerer, 2015;Wingfield & Connell, 2019, Connell, 2019Yee & Thompson-Schill, 2016).…”