Toward a Framework for Interactive Systems to Conduct Digital Audio and Video Streams

We present in this paper a complete gestural interface built to support music pedagogy. The development of this prototype concerned both hardware and software components: a small wireless sensor interface including accelerometers and gyroscopes, and an analysis system enabling gesture following and recognition. A first set of experiments was conducted with teenagers in a music theory class. The preliminary results were encouraging concerning the suitability of these developments in music education. KeywordsTechnology-enhanced learning, music pedagogy, wireless interface, gesture-follower, gesture recognition INTRODUCTIONThe recent developments in the fields of movement analysis and gesture capture technology create appealing opportunities for music pedagogy. For example, traditional instruments can be augmented to provide control over digital musical processes, altering standard instrument practice and offering potentially complementary pedagogical tools. Moreover, the development of novel electronic interfaces/instruments generates even more different paradigms of music performance, giving rise to potential novel approaches in music education.In this article we present a gestural interface that was integrated in a music education context. Both hardware and software components were developed and are described here. First, we report on the design of a relatively inexpensive miniature wireless sensor system that is used with accelerometers and gyroscopes. Second, we describe a gesture analysis system programmed in the Max/MSP environment to perform gesture recognition and following. The complete prototype enables us to experiment with various pedagogical scenarios. This research is currently conducted in the framework of the European I-MAESTRO project on technology enhanced learning, focusing on music education [23].The motivation for this work is grounded in our pedagogical approach that considers physical gesture [11] as a central element for performance but also for the embodiment of music concepts and theory. Our working hypothesis is that specific gestrual interactive systems can enhance this pedagogical approach. Even if similar or complementary tools have been already proposed and carried out [7][9][10][13] [14][27], the use of digital technology and gestural interfaces in music pedagogy is at its very beginning. Any use of new technology in music education represents difficult challenges, nevertheless we believe that such an approach offers great potential. This paper is divided in three separate parts. The first two parts concern the technological developments, respectively the wireless sensor interface and the gesture follower/recognizer. In the third part, we present the pedagogical scenarios and the preliminary results we obtained after a first set of trials in music classes. WIRELESS INTERFACE AND SENSORS 2.1 RequirementsWe developed and reported previously on several wireless interfaces, that were used in applications including the augmented violin project [3] and dance performances [8]. The experienc...

show abstract

“…Similar approaches have been reported [6][12] [13] [14][15] [16] and are often used in implicit mapping strategies.…”

Section: Realtime Gesture Analysis 31 Gesture Following/recognitionsupporting

confidence: 61%

Wireless sensor interface and gesture-follower for music pedagogy

Bevilacqua

Guédy

Schnell

et al. 2007

Proceedings of the 7th International Conference on New Interfaces for Musical Expression - NIME '07

View full text Add to dashboard Cite

show abstract

“…The work by Lee, Karrer, and Borchers (2006) is especially relevant to our work in its discussion of synchronization of beats and smooth time-map adjustment. Other work (Katayose and Okudaira 2004;Baba, Hashida, and Katayose 2010) discusses both tempo adjustment and synchronized score display using an architecture similar to some of our partial implementations.…”

Section: A Vision For Human-computer Music Performancementioning

confidence: 99%

Methods and Prospects for Human–Computer Performance of Popular Music

Dannenberg

Gold

Liang

et al. 2014

Computer Music Journal

View full text Add to dashboard Cite

Computers are often used in performance of popular music, but most often in very restricted ways, such as keyboard synthesizers where musicians are in complete control, or pre-recorded or sequenced music where musicians follow the computer's drums or click track. An interesting and yet little-explored possibility is the computer as highly autonomous performer of popular music, capable of joining a mixed ensemble of computers and humans. Considering the skills and functional requirements of musicians leads to a number of predictions about future human-computer music performance (HCMP) systems for popular music. We describe a general architecture for such systems and describe some early implementations and our experience with them.Sound and music computing research has made a tremendous impact on music through sound synthesis, audio processing, and interactive systems. In "high art" experimental music, we have also seen important advances in interactive music performance, including computer accompaniment, improvisation, and reactive systems of all sorts. In contrast, few if any systems can claim to support "popular" music performance in genres such as rock, jazz, and folk. Here, we see digital instruments, sequencers, and audio playback, but not autonomous interactive machine performers. Although the practice of popular music may not be a very active topic for research in music technology, it is arguably the dominant form of live music. For example, in a recent weekly listing of concerts in Pittsburgh, there are 24 "classical" concerts, 1 experimental or electro-acoustic performance, and 98 listings for rock, jazz, open stage, and acoustic music.In this article, we explore the approaches to interactive popular music performance with computers. We present a vision for such systems in the form of predictions about future performance

show abstract

“…Such interfaces are mostly designed for professional musicians and for integrating prerecorded with live performances. The system proposed by Lee et al [12] enables performers to freely manipulate the tempo of the recorded audio and video. Their approach first extracts the user's beats, and then maps these beats to the music rhythm by performing synchronized audio and video time stretching.…”

Section: Related Workmentioning

confidence: 99%

The visual keyboard: Real-time feet tracking for the control of musical meta-instruments

Jean¹,

Albu²

2008

Signal Processing: Image Communication

View full text Add to dashboard Cite

a b s t r a c tThis paper proposes a new perceptual interface for the control of computer-based music production. We address the constraints imposed by the use of musical meta-instruments during live performance or rehearsal by tracking feet motion relatively to a visual keyboard. The visual attribute stands for the fact that, unlike its physical counterpart, our keyboard does not involve any force feedback during key-presses. The proposed tracking algorithm is structured on two levels, namely a coarse level for foot regions, and a fine level for foot tips. Tracking works in real-time and handles efficiently feet regions merging/unmerging due to spatial proximity and cast shadows. The output of the tracking is used for the spatiotemporal detection of key-''press'' events.

show abstract

Toward a Framework for Interactive Systems to Conduct Digital Audio and Video Streams

Cited by 12 publications

References 16 publications

Wireless sensor interface and gesture-follower for music pedagogy

Wireless sensor interface and gesture-follower for music pedagogy

Methods and Prospects for Human–Computer Performance of Popular Music

The visual keyboard: Real-time feet tracking for the control of musical meta-instruments

Contact Info

Product

Resources

About