This study investigates vowel mappings for a voice synthesizer controlled by hand gestures for artistic performance. The vowel targets are on a horizontal plane navigated by the movement of the right hand in front of the performer. Two vowel mappings were explored. In one mapping, the vowels were evenly distributed in a circle to make the vowel targets easier for the performer to find. In the other mapping, the vowels were arranged according to the F2 versus F1 space. Linear hand motions were then made through the vowel space while plotting the formant trajectories. The evenly distributed mapping resulted in formant trajectories that were not monotonic; the F1 and F2 pitch contours varied up and down as the hand carried out the linear motions. This had the unintended result of producing multiple diphthongs. In contrast, the F2 versus F1 mapping enabled the performer to create monotonic formant trajectories and the perception of a single diphthong. The performer found it easier to speak and sing through the system when a single linear hand motion resulted in a single diphthong. [This project was supported by Canada Council for the Arts, Natural Sciences and Engineering Council of Canada, and Media and Graphics Interdisciplinary Centre.]

We describe progress on creating digital ventriloquized actors (DIVAs). DIVAs use hand gestures to synthesize audiovisual speech and song by means of an intermediate conversion of hand gestures to articulator (e.g., tongue, jaw, lip, and vocal chords) parameters of a computational three-dimensional vocal tract model. Our parallel-formant speech synthesizer is modified to fit within the MAX/MSP visual programming language. We added spatial sound and various voice excitation parameters in an easy-to-use environment suitable for musicians. The musician’s gesture style is learned from examples. DIVAs will be used in three composed stage works of increasing complexity performed internationally, starting with one performer initially and culminating in three performers simultaneously using their natural voices as well as the hand-based synthesizer. Training performances will be used to study the processes associated with skill acquisition, the coordination of multiple “voices” within and among performers, and the intelligibility and realism of this new form of audio/visual speech production. We are also building a robotic face and computer graphics face that will be gesture controlled and synchronized with the speech and song. [This project is funded by the Canada Council for the Arts and Natural Sciences and Engineering Research Council, Canada. More information is at: www.magic.ubc.ca/VisualVoice.htm]

What Does A Body Know? is a concert work forDigital Ventriloquized Actor (DiVA) and sound clips. A DiVA is a real time gesture-controlled formant-based speech synthesizer using a Cyberglove ® , touchglove, and Polhemus Tracker ® as the main interfaces. When used in conjunction with the performer's own voice solos and "duets" can be performed in real time. Figure 1: Marguerite Witvoet, DiVA

No abstract