No abstract

The idea of the project reported here was to design a system that builds on touch to enable people with severe hearing impairments to “listen” to music through a process called sensory substitution. The goal was to transform the auditory parameter space into one that is adequate for haptic perception. The approach reported here builds on (i) the design of a haptic display, a tabletop device with 8–24 actuators that can be driven individually, (ii) machine learning algorithms, and (iii) a psychophysical study to determine which music cues can be perceived through touch. The latter was necessary because vibrotactile perception is not yet well understood in the context of music perception. The double-blind study analyzes how vibrotactile stimuli contribute to the perception, cognition, and distinction of sounds in human participants who have been trained versus those who have not. In order to ensure that normal-hearing participants could not hear sounds radiated from the haptic display, sound isolating headphones were used to playback pink noise during the experiment.

Building on previous findings, the project reported here expands on the idea of how the modality of touch contributes to the sensory perception of sound in the presence of vibro-tactile events. The SenseAble1.0, a vibrotactile interface, was developed to transform the auditory parameter space into one that is adequate for haptic perception by the means of eight individually controllable actuators and machine learning algorithms. Due to the process of sensory substitution, people with hearing impairments can use the vibrotactile display to “listen” to music and perceive exterior acoustic stimuli through the sense of touch. A psychophysical pilot study investigated (i) frequency discrimination and pitch direction as well as (ii) interval size recognition of the haptic modality, and (iii) adaptation effects of the somatosensory system. Live performances using the SenseAble as a vibrotactile music system illustrate to what extent musical communication between ensemble members and a hearing-impaired musician is possible. A new prototype with enhanced spatial resolution properties for perceiving vibrotactile stimuli will be introduced as well. It was designed to improve the usability of sensory substitution systems in real and virtual environments and to maximize the efficiency of co-located and telematic performances in terms of musical communication and interaction.

No abstract