Other ways of seeing: From behavior to neural mechanisms in the online “visual” control of action with sensory substitution

Abstract

Vision is the dominant sense for perception-for-action in humans and other higher primates. Advances in sight restoration now utilize the other intact senses to provide information that is normally sensed visually through sensory substitution to replace missing visual information. Sensory substitution devices translate visual information from a sensor, such as a camera or ultrasound device, into a format that the auditory or tactile systems can detect and process, so the visually impaired can see through hearing or touch. Online control of action is essential for many daily tasks such as pointing, grasping and navigating, and adapting to a sensory substitution device successfully requires extensive learning. Here we review the research on sensory substitution for vision restoration in the context of providing the means of online control for action in the blind or blindfolded. It appears that the use of sensory substitution devices utilizes the neural visual system; this suggests the hypothesis that sensory substitution draws on the same underlying mechanisms as unimpaired visual control of action. Here we review the current state of the art for sensory substitution approaches to object recognition, localization, and navigation, and the potential these approaches have for revealing a metamodal behavioral and neural basis for the online control of action.

Designing sensory-substitution devices: Principles, pitfalls and potential1

“…Brown, Simpson and Proulx (2014) found that there is an upper limit to representation of auditory objects due to intrinsic limits of transmission in auditory cortical areas. Investigations of the capacity of the visual system reveal that it has four orders of magnitude greater bandwidth than haptic perception ( Kokjer, 1987 ; Schmidt, 1981 ) while the information capacity of the human ear falls between these two estimates ( Bialek et al, 1993 ; Dudel, 1986 ; Proulx et al, 2016 ; Wurtz & Kandel, 2000 ). Loomis, Klatzky and Giudice (2012) argue that the bandwidth of touch equals blurred vision through filtering of information in the cutaneous system.…”

Are Supramodality and Cross-Modal Plasticity the Yin and Yang of Brain Development? From Blindness to Rehabilitation

“…SSDs have been also employed in blind individuals during more complex tasks such as spatial navigation (Kupers et al, 2010; Chebat et al, 2011, 2015; Proulx et al, 2015; for a review). The ability to navigate the environment is crucial in modern urban life, yet it represents a challenging task for blind subjects, in particular when novel routes have to be learned.…”

The Effect of Irrelevant Environmental Noise on the Performance of Visual-to-Auditory Sensory Substitution Devices Used by Blind Adults

“…Non-invasive rehabilitative approaches, on the other hand, strive to provide the missing sensory information via other, intact, senses. In recent decades, for instance, much research has been conducted using a specific type of noninvasive rehabilitation procedure namely, visual-to-auditory Sensory Substitution Devices (SSDs) which convey the missing visual information through the intact auditory channel via predetermined algorithms that can be learned by the users (Bach-y-Rita et al, 1998;Meijer, 1992;Proulx et al, 2015). It has been shown that, after training, visual-to-auditory SSD users are able to perform a variety of 'visual' tasks such as object recognition and localization (Proulx et al, 2008;Striem-Amit et al, 2012a), as well as navigation in virtual environments (Maidenbaum et al, 2016), among many other tasks (Maidenbaum, Abboud and Amedi, 2014;Ward and Meijer, 2010).…”