Learning to Perceive with a Visuo — Auditory Substitution System: Localisation and Object Recognition with ‘The Voice’

Abstract

We investigated to what extent participants can acquire the mastery of an auditory-substitution-of-vision device ('The vOICe') using dynamic tasks in a three-dimensional environment. After extensive training, participants took part in four experiments. In the first experiment we explored locomotion and localisation abilities. Participants, blindfolded and equipped with the device, had to localise a target by moving a hand-held camera, walk towards the target, and point at it. In the second experiment, we studied the localisation ability in a constrained pointing task. In the third experiment we explored participants' ability to recognise natural objects via their auditory rendering. In the fourth experiment we tested the ability of participants to discriminate objects belonging to the same category. We analysed participants' performance from both an objective and a subjective point of view. The results showed that participants, through sensorimotor interactions with the perceptual scene while using the hand-held camera, were able to make use of the auditory stimulation to obtain the information necessary for locomotor guidance, localisation, and pointing, as well as for object recognition. Furthermore, analysis from a subjective perspective yielded insights into participants' qualitative experience and into the strategies they used to master the device, and thus to pass from a kind of deductive reasoning to a form of immediate apprehension of what is being perceived.

Perception With Compensatory Devices: From Sensory Substitution to Sensorimotor Extension

“…Rather, we will argue that SSDs transform and extend our perceptual capacities. Following Auvray and her colleagues (Auvray et al, 2007a), Lenay and his colleagues (Lenay, Gapenne, Hanneton, Marque, & Genouëlle, 2003), and congruent with a broader view on cognitive enhancement through external devices (e.g., Clark, 2003;Menary, 2006Menary, , 2007, we thus propose that we should speak of sensory extension, supplementation, or transformation, rather than substitution.…”

“…Since their inception in the 1960s various kinds of devices have been developed, tested, and shown to allow their users to behave to some degree as if they possessed the substituted sensory organ. For instance, thanks to visual-to-auditory or visual-to-tactile conversion systems, blind persons are able to localize and recognize objects in three-dimensional space (e.g., Auvray, Hanneton, & O'Regan, 2007a;Bach-y-Rita, Collins, Saunders, White, & Scadden, 1969).…”

Loops, constitution, and cognitive extension

“…In section 2.1 we encountered the problem of long multiplication and in section 3.2 we considered the case of an agent perceiving his environment through a tactile visual substitution system. Less well-known, but equally interesting examples are tactile vestibular substitution systems (Tyler et al 2003), auditory visual substitution systems (e.g., vOICe) (Auvray et al 2007), and magnetic perception (Nagel at al. 2005).…”

The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms

“…headphones, vibrotactile array). Within a few hours of training novice participants have some ability to localise and recognise objects (Auvray, Hanneton, & O'Regan, 2007;Brown, Macpherson, & Ward, 2011) and generalise to new objects (Kim & Zatorre, 2008). Expert blind users recruit 'visual' cortices to process the substituted sense (Amedi et al, 2007;Merabet et al, 2009;Poirier, De Volder, Tranduy, & Scheiber, 2007; cf.…”