Data-driven computational models reveal perceptual simulation in word processing

Abstract

Note: This is the author's preprint version of the article (date: 13. 11. 2020). The final article is accepted for publication in Journal of Memory and Language. In their strongest formulation, theories of grounded cognition claim that concepts are made up of sensorimotor information. Following such equivalence, perceptual properties of objects should consistently influence processing, even in purely linguistic tasks, where perceptual information is neither solicited nor required. Previous studies have tested this prediction in semantic priming tasks, but they have not observed perceptual influences on participants' performances. However, those findings suffer from critical shortcomings, which may have prevented potential visually grounded/perceptual effects from being detected. Here, we investigate this topic by applying an innovative method expected to increase the sensitivity in detecting such perceptual effects. Specifically, we adopt an objective, data-driven, computational approach to independently quantify vision-based and language-based similarities for prime-target pairs on a continuous scale. We test whether these measures predict behavioural performance in a semantic priming mega-study with various experimental settings. Vision-based similarity is found to facilitate performance, but a dissociation between vision-based and language-based effects was also observed. Thus, in line with theories of grounded cognition, perceptual properties can facilitate word processing even in purely linguistic tasks, but the behavioural dissociation at the same time challenges strong claims of sensorimotor and conceptual equivalence.

Out-of-vocabulary but not meaningless: Evidence for semantic-priming effects in pseudoword processing.

“…These results may indicate that the semantic effect for pseudowords may be the result of automatic processes operating at shorter SOAs, while the effect may disappear at longer SOAs when controlled processing strategies become activated (e.g., K€ uper & Heil, 2009). This is consistent with previous studies showing that weaker effects in semantic priming might be associated with faster activation decay, and hence are less likely to be observed in long-SOA paradigms (e.g., Petilli et al, 2021).…”

ViSpa (Vision Spaces): A computer-vision-based representation system for individual images and concept prototypes, with large-scale evaluation.

“…A more sophisticated account would instead move from the actual stimuli experienced by humans (light of a certain wavelength) and build on how they are processed by the cognitive system in order to explain behavior. In fact, it should ultimately be the aim of our scientific endeavor to model mental representations starting from the input level —the arrangements of stimuli that humans experience, and from which they form their representations (Petilli et al, 2021; Westbury, 2016). One highly influential class of psychological models adopting this principle is the connectionist models developed in the 1980s (McClelland et al, 1986; Rumelhart et al, 1986), which adapted and further developed the neural-network models that nowadays dominate computer science and artificial intelligence research (especially since the arrival of deep learning models; LeCun et al, 2015).…”

The Flickr frequency norms: what 17 years of images tagged online tell us about lexical processing

“…Moreover, although vision and language are frequently treated as distinct domains in the literature, they are entangled in our experience in such a way that one ends up influencing the other (Cohn & Schilperoord, 2022). On the one hand, there are several studies which highlight a clear impact of visual experience on language: asymmetries in our perceptual experiences are reflected in our vocabulary (e.g., Winter et al, 2018); and even in purely linguistic contexts, the visual properties of objects affect conceptual processing (e.g., Günther et al, 2020;Petilli et al, 2021;Zwaan et al, 2002). Likewise, measures pertaining to the visual experience with a word referent -such as ratings of concreteness (i.e., how concrete vs abstract a XXX is), imageability (i.e., how easy it is to form an image of XXX) or visual strength (i.e., to what extent do you experience XXX by seeing) -proved to be important predictors of word processing speed (Binder et al, 2005;Bleasdale, 1987;Connell & Lynott, 2012De Groot, 1989;Lynott et al, 2020;Vergallito et al, 2020).…”

“…Chen & Gupta, 2015;Das & Clark, 2018). Furthermore, Flickr images constitute a quite large part of ImageNet (Deng et al, 2009), a large-scale database of labelled images adopting the hierarchical category structure of WordNet (Miller, 1998) and designed for use in visual object recognition research (e.g., Krizhevsky et al, 2012), but also adopted to build up prototypical vision-based representations for concepts to be used in psychological research (Anderson et al, 2015;Günther et al, 2021;Petilli et al, 2021).…”