Perceptual Similarity: Insights From Crossmodal Correspondences

Abstract

Perceptual similarity is one of the most fiercely debated topics in the philosophy and psychology of perception. The documented history of the issue spans all the way from Plato – who regarded similarity as a key factor for human perceptual experience and cognition – through to contemporary psychologists – who have tried to determine whether, and if so, how similarity relationships can be established between stimuli both within and across the senses. Recent research on cross-sensory associations, otherwise known as crossmodal correspondences – that is, the existence of observable consensual associations, or mappings, between stimuli across different senses – represents an especially interesting field in which to study perceptual similarity. In fact, most accounts of crossmodal association that have been put forward in the literature to date evoke perceptual similarity as a key explanatory factor mediating the underlying association. At the same time, however, these various accounts raise several important theoretical questions concerning the very nature of similarity, with, for example, the sensory, affective, or cognitive underpinnings of similarity judgements remaining unclear. We attempt to shed light on these questions by examining the various accounts of crossmodal associations that have been put forward in the literature. Our suggestion is that perceptual similarity varies from being phenomenologically-based to conceptually-based. In particular, we propose that the nature of the associations underlying similarity judgements – whether these associations are phenomenologically-, structurally-, emotionally-, or conceptually-based – may be represented in a two-dimensional space with associative strength on one axis, and cognitive penetrability on the other.

Crossmodal associations between naturally occurring tactile and sound textures

“…To understand the nature of these crossmodal associations, we interpret the qualitative results from the perspective of the perception of similarity. The concept of similarity is a fundamental aspect of perception (Goldstone & Barsalou, 1998; Tversky, 1977), and there are multiple ways for the perception of similarity between stimuli of two modalities to occur (Di Stefano & Spence, 2023). It can range from a similarity in phenomenological experience to the conceptual connection between the stimuli.…”

“…When two stimuli from different modalities are reported to have an association, it could be because of the perception of similarity between them. This perception of similarity could happen for multiple reasons: phenomenological similarity, amodal property, relative positioning, common affective quality, analogical reasoning, or statistical learning (Di Stefano & Spence, 2023). In our study, we show that the crossmodal associations between sound and tactile textures show a variety of strategies, mainly emotional mediation and cognitive strategies.…”

Marketing sonified fragrance: Designing soundscapes for scent

“…Here, though, another potentially important challenge arises, namely that the temporal rate of information processing in olfaction is simply much lower than it is in audition (see Gallace et al, 2012 ). Finally, it should be noted that researchers have long questioned whether it is even possible to experience any kind of perceptual similarity between the impressions that are available to the different senses (see Di Stefano & Spence, 2023 ). As such, the best that might ultimately be possible in terms of crossmodal matching is some kind of affective alignment of emotional sequences of sensations attached with the various notes and scents.…”