Length and orientation constancy learning in 2-dimensions with auditory sensory substitution: the importance of self-initiated movement

Abstract

A subset of sensory substitution (SS) devices translate images into sounds in real time using a portable computer, camera, and headphones. Perceptual constancy is the key to understanding both functional and phenomenological aspects of perception with SS. In particular, constancies enable object externalization, which is critical to the performance of daily tasks such as obstacle avoidance and locating dropped objects. In order to improve daily task performance by the blind, and determine if constancies can be learned with SS, we trained blind (N = 4) and sighted (N = 10) individuals on length and orientation constancy tasks for 8 days at about 1 h per day with an auditory SS device. We found that blind and sighted performance at the constancy tasks significantly improved, and attained constancy performance that was above chance. Furthermore, dynamic interactions with stimuli were critical to constancy learning with the SS device. In particular, improved task learning significantly correlated with the number of spontaneous left-right head-tilting movements while learning length constancy. The improvement from previous head-tilting trials even transferred to a no-head-tilt condition. Therefore, not only can SS learning be improved by encouraging head movement while learning, but head movement may also play an important role in learning constancies in the sighted. In addition, the learning of constancies by the blind and sighted with SS provides evidence that SS may be able to restore vision-like functionality to the blind in daily tasks.

Perception of Graphical Virtual Environments by Blind Users via Sensory Substitution

“…Participants experienced the objects in the scenes from different distances and angles, in which their visual representation and corresponding auditory representation were very different ( Fig 6 ), making the task of maintaining object constancy and the mental map of their environment even harder. We suggest their successful results may be linked with the combination of active sensing, gamification and user immersion which have all been suggested in the past as potential SSD training boosters [ 19 , 22 , 30 , 44 , 45 ].…”

“…SSDs are non-invasive interfaces which translate information from one sense into another, here from vision into audition [ 21 ]. These devices have been used extensively for psychophysical [ 17 , 19 , 22 , 23 ] and neuroimaging [ 24 – 27 ] research and to a more limited extent also for practical use in the real world [ 28 , 29 ]. Their transfer to practical use still faces some significant challenges, and though many of these have been mitigated in recent years one of the main remaining challenges is in training and mastering their use (reviewed at length in [ 30 , 31 ]).…”

Improving training for sensory augmentation using the science of expertise

“…Input from SSADs can drive subcortically supported behaviour such as visual saccades in the absence of conscious understanding (Wright et al, 2012), but is susceptible to top-down conscious influence (Murphy et al, 2016). As experience with a device progresses, processing the stimuli involves areas of cortex involved in higher level feature processing and identification (StriemAmit et al, 2012a;Striem-Amit & Amedi, 2014). SSAD have been suggested as a controlled method of studying cross-modal plasticity following sensory impairment and a potential biomarker for adaptability to more invasive vision restoration technologies (Nau et al, 2015a).…”

“…As assistive technology, modern SSADs are applied to activities of daily life such as reading, object recognition, and navigation of the environment (Striem-Amit et al, 2012a;Maidenbaum et al, 2016;Nau et al, 2015b). A substantial number of SSADs focus on providing visual information, and SSADs have been presented as an alternative to retinal implants as a method for restoring visual function (Striem-Amit et al, 2012b).…”

“…There are factors that might influence an appropriate duration; for example, the complexity of device output may be an influencing factor in the length of time needed for users to pick up the basics of the device, or to develop a fuller understanding of its capabilities. Isolating objects from the background and identifying them in the output provided by visual to auditory and visual to tactile SSADs may involve extensive training (Striem-Amit et al, 2012a;Nau et al, 2015b;Lee et al, 2014). In contrast, when the stimulus is more simple, e.g.…”