On the use of the auditory pathway to represent image scenes in real-time

Abstract

The See Color interface transforms a small portion of a coloured video image into sound sources represented by spatialised musical instruments. Basically, the conversion of colours into sounds is achieved by quantisation of the HSL colour system. Our purpose is to provide visually impaired individuals with a capability of perception of the environment in real time. In this work we present the system principles of design and several experiments that have been carried out by several blindfolded persons with See ColOr prototypes related to static pictures on a tablet and simple video images. The goal of the first experiment was to identify the colours of static pictures' main features and then to interpret the image scenes. Although learning all instrument sounds in only a training session was too difficult, participants found that colours were helpful to limit the possible image interpretations. The experiments on the analysis of static pictures suggested that the order of magnitude of the slow down factor related to the use of the auditory channel, instead of the visual channel could correspond to the order of magnitude related to the ratio of visual channel capacity to auditory channel capacity. Afterwards, two experiments based on a head mounted camera have been performed. The first experiment pertaining to object manipulation is based on the pairing of coloured socks, while the second experiment is related to outdoor navigation with the goal of following a coloured serpentine painted on the ground. The "socks" experiment demonstrated that blindfolded individuals were able to accurately match pairs of coloured socks. The same participants with the addition of a blind individual successfully followed a red serpentine painted on the ground for more than 80 meters. According to task time durations, the order of magnitude of the slow down factor related to the "socks" and "serpentine" experiments could be equal to one. From a cognitive perspective this would be consistent with the fact that these two tasks are simpler than the interpretation of image scenes.

Exploration and avoidance of surrounding obstacles for the visually impaired

“…In order to reduce the time spent to capture a large field of view, digital cameras are employed, such as EYECane [14], See ColOr [3], and vOICe [19]. Nevertheless, it is challengeable that the digital cameras locate obstacles with precise distance and orientation.…”

Software Design for Low-Latency Visuo-Auditory Sensory Substitution on Mobile Devices

“…SeeColOr (Bologna et al, 2009a;Deville et al, 2009;Bologna et al, 2009b;Bologna et al 2010) applies image filters to a stereoscopic colored video acquisition with a Hue-Saturation-Luminance (HSL) color format. A method, based on the combination of several conspicuity maps obtained from the extraction of different features, generates a simplified video stream that is afterwards sonified.…”

Hearing colors: an example of brain plasticity

“…However, these devices often generate unpleasant sensations and lack color information. This motivated the creation of new SSDs that encode color information such as Soundview (Nie et al, 2009 ), Haptic-Color Glove (Kahol et al, 2006 ), SeeColOr (Bologna et al, 2009 ) and EyeMusic (Abboud et al, 2014 ).…”