Vibrotactile music systems for co-located and telematic performance

Abstract

Building on previous findings, the project reported here expands on the idea of how the modality of touch contributes to the sensory perception of sound in the presence of vibro-tactile events. The SenseAble1.0, a vibrotactile interface, was developed to transform the auditory parameter space into one that is adequate for haptic perception by the means of eight individually controllable actuators and machine learning algorithms. Due to the process of sensory substitution, people with hearing impairments can use the vibrotactile display to “listen” to music and perceive exterior acoustic stimuli through the sense of touch. A psychophysical pilot study investigated (i) frequency discrimination and pitch direction as well as (ii) interval size recognition of the haptic modality, and (iii) adaptation effects of the somatosensory system. Live performances using the SenseAble as a vibrotactile music system illustrate to what extent musical communication between ensemble members and a hearing-impaired musician is possible. A new prototype with enhanced spatial resolution properties for perceiving vibrotactile stimuli will be introduced as well. It was designed to improve the usability of sensory substitution systems in real and virtual environments and to maximize the efficiency of co-located and telematic performances in terms of musical communication and interaction.