Personal orchestra: a real-time audio/video system for interactive conducting

Abstract

We present the first multimedia system to conduct a realistic electronic orchestra. Users can control tempo, dynamics, and instrument emphasis of the orchestra through natural conducting gestures with an infrared baton. Using gesture recognition and tempo adjustment algorithms, the system plays back an audio and video recording of an actual orchestra that follows the user's conducting in real time. A major achievement of this system is its ability to vary playback speed in real time while avoiding audio artifacts such as pitch changes. The system has been deployed as an exhibit and has become a major attraction of a large Vienna-based music exhibition center.

A Mobile Conducting App With a Switching Beat-Following Algorithm

“…Music conducting systems have appeared in various forms: multimedia installations, such as Personal Orchestra (Borchers et al, 2004), Mendelssohn Effektorium (WHITEvoid Studio, 2014); computer programs, such as Semi-conductor (Google Creative Lab, 2018), Live Tempo of Avid Sibelius (Avid Technology, 2023); and mobile apps, such as Bravo Gustavo (Hello Design, 2010), vMaestro (Lim & Yeo, 2014), Airconductor (Micro App Brewery, 2019). All these systems have three major components: a music player, a beat tracker, and a beat-following algorithm to let the music follow the beats.…”

COSMOS: Computational Shaping and Modeling of Musical Structures

“…Many use actual conducting gestures, or archetypal forms of the gestures, to control audio playback; a number of such systems exist as museum installations. In the Personal Orchestra (Borchers et al, 2004 ) at Vienna's Haus der Musik, conducting motions control the video and audio playback of a performance by the Vienna Philharmonic. The Mendelssohn Effektorium at Leipzig's Mendelssohn-Bartholdy Museum lets the user conduct, using a baton, an orchestra of free standing speakers each representing an orchestra section; the setup affords the conductor loudness control over individual orchestra sections.…”

Interacting with a Virtual Conductor

“…Many systems have been made that try to follow a human conductor. They use, for example, a special baton [3], a jacket equipped with sensors [4] or webcams [5] to track conducting movements. Strategies to recognize gestures vary from detecting simple up and down movements [3] through a more elaborate system that can detect detailed conducting movements [4] to one that allows extra system-specific movements to control music [5].…”

“…They use, for example, a special baton [3], a jacket equipped with sensors [4] or webcams [5] to track conducting movements. Strategies to recognize gestures vary from detecting simple up and down movements [3] through a more elaborate system that can detect detailed conducting movements [4] to one that allows extra system-specific movements to control music [5]. Most systems are built to control the playback of music (MIDI or audio file) that is altered in response to conducting slower or faster, conducting a subgroup of instruments or conducting with bigger or smaller gestures.…”