Designing, Implementing and Evaluating a Cross-Modal Sensory Substitution System for the Effective Communication of the Emotional and Informative Aspects of Music

Abstract

Visualizations have existed for millennia as ways to communicate information. Visualizations are ubiquitous tools used every day to help navigate cities and aid in learning complex tasks. Tasks are made simpler when applying various visualization methods to large data sets to discover trends that are otherwise difficult to notice. More recently, music visualization systems have been created to convey music in the visual domain; however, they are not based on any psychological model of auditory and visual equivalents. This thesis discusses a music visualization system called MusicViz, which facilitates in the visual communication of the informative and entertainment aspect of music based on psychologically justified translation principals. MusicViz is combined with a vibro-feedback chair called the Emoti-Chair, which translates auditory music to vibrations along the user’s back. The combined system is coined VITA (Visually Immersive and Tactile Animation). A usability evaluation of the VITA showed it is an enjoyable experience.