An integrated approach to motion and sound

Abstract

SUMMARYUntil recently, sound has been given little attention in computer graphics and related domains of computer animation and virtual environments, although sounds which are properly synchronized to motion provide a great deal of information about events in the environment. Sounds are often not properly synchronized because the sounds and the phenomena that caused the sounds are not considered in an integrated way. In this paper, we present an integrated approach to motion and sound as it applies to computer animation and virtual environments. The key to this approach is synchronization by mapping the motion parameters to sound parameters so that the sound changes as a result of changes in the motion. This is done by representing sounds using a technique for functional composition analogous to the "shade trees" which we call timbre trees. These timbre trees are used as a part of a sound description language that is analogous to scene description languages like RenderMan.Using this methodology, we have produced convincing sound effects for a wide variety of animated scenes including the automatic generation of background music.

Interactive Simulation of rigid body interaction with friction-induced sound generation

“…At present it is still common practice to add sound effects by hand onto film shots, even though automatic soundtrack production and synchronization for computer animations was proposed about a decade ago [3], [4]. In those works, prototype sounds, either recordings or synthesis models, are attached to objects and activated by sound event scripts.…”

“…The animations represent examples of everyday friction sounds, and have been designed using external graphical libraries of pd, specifically the OpenGL-based gem. 4 In all the examples the main sound generation mechanism is the stick-slip relative motion.…”

“…As in the previous example, the rubbing finger is controlled through mouse input. Interestingly, 4 http://gem.iem.at/ setting the glass into resonance is not a trivial task and requires some practice and careful control, just as in the real world.…”

Collaborative Computer-Aided Parameter Exploration for Music and Animation

“…Of particular note is the work of Hahn et al [9,29], where visual and acoustic events are generated together. Their approach seeks to synchronize the development of sound and motion by connecting the sound and motion parameters before rendering.…”

Music-driven motion editing: local motion transformations guided by music analysis

“…In these systems the motion parameters are tied to parameterisable musical constructs such as Timbre-trees [10], C-Sound models, or music generators [19]. The quality of the soundtrack is then directly related to the effectiveness of the sound representation.…”