Crossmodal processing and sensory substitution: Is “seeing” with sound and touch a form of perception or cognition?

Abstract

The brain has evolved in this multisensory context to perceive the world in an integrated fashion. Although there are good reasons to be skeptical of the influence of cognition on perception, here we argue that the study of sensory substitution devices might reveal that perception and cognition are not necessarily distinct, but rather continuous aspects of our information processing capacities.

Multisensory inclusive design with sensory substitution

“…The potential exists for the use of sensory substitution techniques to benefit a wide range of users. As sensory substitution stands between perception and cognition (Arnold, Pesnot-Lerousseau, & Auvray, 2017;Esenkaya & Proulx, 2016), exploring sensory substitution phenomena in a broader multisensory context could contribute new insights into how different sensory information and forms are interconnected with each other via cognitive forms. If we can better understand the ways in which sensory substitution techniques work, we may be able to better support individuals who have disabilities by developing inclusive technologies.…”

“…2 ). The representation of features of a sensory experience, such as seeing, using a different sensory form, such as hearing, is called sensory substitution (Esenkaya & Proulx, 2016 ). Accordingly, sensory substitution techniques take advantage of cross-modal correspondences by evoking in the brain a cognitive form by stimulating a different sense from the one usually stimulated.…”

“…The mappings between visual and auditory forms, in terms of elevation and pitch, and brightness and loudness, can be utilised via SSDs to represent some features of a visual form with an auditory form (Meijer, 1992 ). In the long term, these pairings may be strongly associated such that late blind people can have visual imagery similar to that of the perception of sight (Esenkaya & Proulx, 2016 ; Ortiz et al, 2011 ; Ward & Meijer, 2010 ). With SSDs, it is possible to acquire visual information by means of sonifications (Meijer, 1992 ) or two-dimensional tactile cues (Bach-y-Rita & W. Kercel, 2003 ).…”

Inclusive Education Technologies

“…How do we leverage cognition and perception research to maximise educational benefits for people with visual impairments? Sensory substitution and cross-modal studies contribute to our understanding of how the visual parts of the brain process other information in the absence of visual input [7,10], digital interactive tools that introduce novelty and stimulate activity can play a powerful role in creating new neural pathways to support learning processes [16,27], and usability studies of crossmodal tools inform us about how to improve multisensory user expeirences [24,11].…”