Seeing ‘Where’ through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution

Abstract

BackgroundSensory substitution devices for the blind translate inaccessible visual information into a format that intact sensory pathways can process. We here tested image-to-sound conversion-based localization of visual stimuli (LEDs and objects) in 13 blindfolded participants.Methods and FindingsSubjects were assigned to different roles as a function of two variables: visual deprivation (blindfolded continuously (Bc) for 24 hours per day for 21 days; blindfolded for the tests only (Bt)) and system use (system not used (Sn); system used for tests only (St); system used continuously for 21 days (Sc)). The effect of learning-by-doing was assessed by comparing the performance of eight subjects (BtSt) who only used the mobile substitution device for the tests, to that of three subjects who, in addition, practiced with it for four hours daily in their normal life (BtSc and BcSc); two subjects who did not use the device at all (BtSn and BcSn) allowed assessment of its use in the tasks we employed. The impact of long-term sensory deprivation was investigated by blindfolding three of those participants throughout the three week-long experiment (BcSn, BcSn/c, and BcSc); the other ten subjects were only blindfolded during the tests (BtSn, BtSc, and the eight BtSt subjects). Expectedly, the two subjects who never used the substitution device, while fast in finding the targets, had chance accuracy, whereas subjects who used the device were markedly slower, but showed much better accuracy which improved significantly across our four testing sessions. The three subjects who freely used the device daily as well as during tests were faster and more accurate than those who used it during tests only; however, long-term blindfolding did not notably influence performance.ConclusionsTogether, the results demonstrate that the device allowed blindfolded subjects to increasingly know where something was by listening, and indicate that practice in naturalistic conditions effectively improved “visual” localization performance.

Seeing with Sound? Exploring Different Characteristics of a Visual-to-Auditory Sensory Substitution Device

“…In principle, the body part itself versus its mode of exploration could be studied independently (eg by moving the head in a hand-like way and moving the hand in a head-like way). In addition, it is important that future research develops paradigms for assessing localisation of objects with sensory substitution devices using measures of reaching and grasping (eg as in Proulx et al 2008) which may depend less on consciously reportable knowledge of position (eg Goodale and Milner 1992).…”

The Enactive Torch: A New Tool for the Science of Perception

“…And, indeed, the idea that the experience of TVSSuse is like vision is supported by evidence for similar perceptual effects, e.g., occlusion and parallax, as well as by verbal reports of some of the participants. In addition, a recent study by Ward and Meijer [53] into the long-term effects of using a related visual-to-sound substitution system the "vOICe," see [38] in blind users has provided strong indications that visual-like experiences are in fact made possible by these kinds of interfaces see also [44].…”

Mobile Video-to-Audio Transducer and Motion Detection for Sensory Substitution

“…The ability to localize static objects has already been demonstrated (Proulx et al, 2008). Other works have shown that participants are capable of grabbing a distant object (Levy-Tzedek et al, 2012).…”