Visual Echolocation Concept for the Colorophone Sensory Substitution Device Using Virtual Reality

Abstract

Detecting characteristics of 3D scenes is considered one of the biggest challenges for visually impaired people. This ability is nonetheless crucial for orientation and navigation in the natural environment. Although there are several Electronic Travel Aids aiming at enhancing orientation and mobility for the blind, only a few of them combine passing both 2D and 3D information, including colour. Moreover, existing devices either focus on a small part of an image or allow interpretation of a mere few points in the field of view. Here, we propose a concept of visual echolocation with integrated colour sonification as an extension of Colorophone—an assistive device for visually impaired people. The concept aims at mimicking the process of echolocation and thus provides 2D, 3D and additionally colour information of the whole scene. Even though the final implementation will be realised by a 3D camera, it is first simulated, as a proof of concept, by using VIRCO—a Virtual Reality training and evaluation system for Colorophone. The first experiments showed that it is possible to sonify colour and distance of the whole scene, which opens up a possibility to implement the developed algorithm on a hardware-based stereo camera platform. An introductory user evaluation of the system has been conducted in order to assess the effectiveness of the proposed solution for perceiving distance, position and colour of the objects placed in Virtual Reality.

Colorophone 2.0: A Wearable Color Sonification Device Generating Live Stereo-Soundscapes—Design, Implementation, and Usability Audit

“…Importantly, the system's implementation does not provide information about the distance to the objects. However, this issue has been addressed by developing an additional Colorophone visual echolocation function introduced in [81].…”

The Topo-Speech sensory substitution system as a method of conveying spatial information to the blind and vision impaired

“…Human echolocators make clicking sounds with their tongues and carefully listen to the echoes reverberating back to them from the objects in their surroundings. New technologies incorporate an element of color into echolocation-inspired devices, such as the EyeMusic ( Abboud et al, 2014 ) and the Colorphone ( Bizoń-Angov et al, 2021 ), which also incorporates a dimension of depth.…”

Investigating Colour-Sound Mapping in Children and Adults: A Pilot Study

“…The literature on the topic of sonification of color is rich and diverse, covering a wide range of subjects from the design of auditory color spaces [1] to the development of virtual reality tools for training and evaluation [2]. The field has seen the creation of various devices such as the ColorPhone [3], the Sensory Substitution Device [1], and the See ColOr [4], as well as studies on the design of sonification methods [5].…”