Auditory coding of visual patterns for the blind

Abstract

Recognition tasks of simple visual patterns have been used to assess an early visual--auditory sensory-substitution system, consisting of the coupling of a rough model of the human retina with an inverse model of the cochlea, by means of a pixel-frequency relationship. The potential advantage of the device, compared with previous ones, is to give the blind the ability to both localise and recognise visual patterns. Four evaluation sessions assessed the performance of twenty-four blindfolded sighted subjects using the device. Subjects had to recognise twenty-five visual patterns, one at a time, using a head-mounted small camera and interpreting the corresponding sounds given by the device. Half the subjects were trained by means of a correction feedback procedure during ten one-hour training sessions embedded in between the evaluation sessions. Results revealed extremely successful training effects. Performance of trained subjects significantly increased with practice compared with the untrained control group. The improvement was also observed for new patterns, demonstrating a learning-process generalisation. The negative correlation observed between scores and processing time showed that the subjects' response accuracy was related to their speed. In conclusion, simple pattern recognition is possible with a fairly natural vision-to-audition coding scheme, given the possibility for the subjects to have sensory--motor interactions while using the device.

Auditory substitution of vision: pattern recognition by the blind

“…The instrumentation used in this study was based on the one of Capelle et al (1998) and Arno et al (1999). The original PSVA set-up consists of a TV camera connected to a PC including a frame grabber and a sound generation printed circuit board, itself connected to an audio ampli®er and headphones (Capelle et al, 1998).…”

“…In a previous experiment (Capelle et al, 1998;Arno et al, 1999), we have shown that trained normal blindfolded subjects can recognize simple visual patterns using a Prosthesis Substituting Vision by Audition (PSVA). The user carried on the head a miniature TV camera and a pair of headphones, both connected to a PC that simultaneously coded into sounds the images captured by the TV camera.…”

“…In our previous experiment, subjects used head movements to scan the visual environment (Capelle et al, 1998;Arno et al, 1999). In the present study, patterns were displayed on a PC screen.…”

Software Design for Low-Latency Visuo-Auditory Sensory Substitution on Mobile Devices

“…The first system of this type is called Prosthesis Substituting Vision with Audition (PSVA) (Capelle et al, 1998;Arno et al, 1999Arno et al, , 2001. The main differences between this project and the previous one are the following: an online sonification without left-to-right scans and frequency encoding depending on both the horizontal and vertical positions of the pixels.…”

Mobile Video-to-Audio Transducer and Motion Detection for Sensory Substitution

“…Another system that started slightly later is called Prosthesis Substituting Vision with Audition (PSVA) (Capelle et al, 1998;Arno et al, 1999Arno et al, , 2001. The main differences with this project compared to the previous one are an on-line sonification without left-to-right scans, and frequency encoding depending on both horizontal and vertical positions of the pixels.…”