<title>Using sound to extract meaning from complex data</title>

Abstract

In analyzing abstract data sets, it is useful to represent them in several alternative formats, each format bringing out different aspects of the data. While most work in data mapping has focused on visual representations, we have found that sonic representations can also be effective aids in interpreting complex data, especially when sonification is used in conjunction with visualization.We have developed prototypes for several high-level sonification tools that can be applied to a wide variety of data. While we used programmable multi-processor digital signal processing hardware to develop and experiment with these prototypes, each of these tools could be implemented as special-purpose hardware or software for use by scientists in specific applications. Our prototype tools include: Mapper (maps data to various sonic parameters), Comparator (feeds a different mapping into each speaker channel), Sonic Histogram (maps the magnitude of each category onto the amplitude of its associated sound), Shifter (shifts signals into the audible range), Marker (a sonic alarm that marks a specific condition).We tested these tools by using them to generate data-driven sound tracks for video animations generated from the same data. The resulting videos provide an increased data bandwidth and an increased sense of virtual reality to the viewer. INTRODUCTION Defining the problemOur goal is to transfer information between two sophisticated, specialized processors; one is electronic and of recent design, and the other is biological and has been undergoing strenuous testing and refmement for several million years. Modelling and analysis programs on the electronic computer are capable of generating vast amounts of data, though not necessarily in an optimal input form for the biological computer. Our job is to structure this data in ways that will improve the performance of the human observer in interpretation and analysis.There are five input ports to the biological processor: the eyes, the ears, the nose, the tongue and the skin. The first of these perceptual modes, vision, has received the most attention. We decided to investigate the auditory system and its suitability as an additional input channel. Why use sound?There are several reasons for investigating the use of sound in data interpretation and analysis:. It adds additional bandwidth to a purely visual interface . It increases the user's sense of engagement with the model world S People have come to expect sound to accompany animated images S The auditory system excels at certain tasks 2.2.1. Bandwidth, engagement, and expectationThe auditory system can provide an alternative input channel alleviating visual overload or clutter.1'2 Visual and auditory representations can be used together to reinforce information and provide a greater sense of engagement with the model world.23Our initial project was to add data-driven sound to videos of scientific visualizations generated at NCSA. In the context of watching a video, people have come to expect a sound track and may even feel un...

<title>Microassembly using auditory display of force feedback</title>

“…Rossister and Wai-Yin Ng [11] had developed a sound device, which allows them to identify the anatomical tissue of a threedimensional human head virtual representation. The frequency of the sound changes with the density of the material in contact, the intensity is related to a relative position and different sounds are defined for different parts (hair, fat, soft tissue, bone...).…”

Exploring geo-scientific data in virtual environments

“…Sounds have proven effective as a way to represent multi-variate data in studies with botanical classification [15], monitoring a patient during surgery [16], the analysis of sociological data [13], atmospheric chemistry [4], and debugging parallel computer programs [12]. The potential for real-time interactive sonification in virtual environments was demonstrated with chaotic data at the NCSA [9].…”

A multimodal data collection tool using REALbasic and Mac OS X

“…There has been some prior work on sonified graphs, using simulated data sets with different types of graphs (Flowers, Buhman, & Turnage, 1997;Flowers & Hauer, 1992, 1995, whereas other researchers have focused on sonified graphs of complex data sets in research settings (Blattner, Greenberg, & Kamegai, 1990;Lunney & Morrison, 1990;Scaletti & Craig, 1990). However, there has been little research to date in which the use of sound, in addition to visual displays, for real data sets that are used in educational settings has been investigated.…”