Online Gesture Analysis and Control of Audio Processing

Bevilacqua, Frédéric; Schnell, Norbert; Rasamimanana, Nicolas; Zamborlin, Bruno; Guédy, Fabrice

doi:10.1007/978-3-642-22291-7_8

Cited by 28 publications

(24 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This mapping allows the performer to reshape a sound based on the temporal morphology of his or her gesture. The general concept of temporal mapping was previously introduced by Bevilacqua et al (2011) for the cases where temporal relationships between gesture and sound parameter profiles are established.…”

Section: Discussionmentioning

confidence: 99%

Mapping Through Listening

Caramiaux

Françoise

Schnell

et al. 2014

Computer Music Journal

Self Cite

View full text Add to dashboard Cite

Gesture-to-sound mapping is generally defined as the association between gestural and sound parameters. This article describes an approach that brings forward the perception-action loop as a fundamental design principle for gesture-sound mapping in digital music instrument. Our approach considers the processes of listening as the foundation-and the first step-in the design of action-sound relationships. In this design process, the relationship between action and sound is derived from actions that can be perceived in the sound. Building on previous work on listening modes and gestural descriptions, we propose to distinguish between three mapping strategies: instantaneous, temporal, and metaphorical. Our approach makes use of machine-learning techniques for building prototypes, from digital music instruments to interactive installations. Four different examples of scenarios and prototypes are described and discussed.

show abstract

Section: Discussionmentioning

confidence: 99%

Mapping Through Listening

Caramiaux

Françoise

Schnell

et al. 2014

Computer Music Journal

Self Cite

View full text Add to dashboard Cite

show abstract

“…In comparison with the template-based model introduced in [1], the multimodal HMM tolerates larger variation occurring between the performance and the training data. Precisely, an evaluation of the algorithm on synthetic data indicates a better ability of the multimodal HMM to capture gesture variations in relationship with sound variations.…”

Section: Discussionmentioning

confidence: 99%

“…Gesture recognition methods often model the temporal dynamics of gestures but are usually limited to classification, restricting mapping strategies to discrete interactions such as triggering [7,5]. The temporal mapping method we recently proposed can learn the temporal relationship between a gesture and a sound using continuous gesture recognition and alignment [1]. Nevertheless, the method is limited to modeling the gesture only.…”

Section: Gesture-sound Relationships In Digital Musical Instrumentsmentioning

confidence: 99%

A multimodal probabilistic model for gesture--based control of sound synthesis

Françoise

Schnell

Bevilacqua

2013

Proceedings of the 21st ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a multimodal approach to create the mapping between gesture and sound in interactive music systems. Specifically, we propose to use a multimodal HMM to conjointly model the gesture and sound parameters. Our approach is compatible with a learning method that allows users to define the gesture-sound relationships interactively. We describe an implementation of this method for the control of physical modeling sound synthesis. Our model is promising to capture expressive gesture variations while guaranteeing a consistent relationship between gesture and sound.

show abstract

“…The model has been first developed for synchronization of the gesture onto a musical score. Subsequently, it has been used as continuous parameter for expressive musical interaction [2]. An extension proposed by Françoise et al [8] takes into account switches between gestures drawing upon a structure of hierarchical HMM.…”

Section: Related Workmentioning

confidence: 99%

Motion Modeling for Expressive Interaction

Caramiaux

2014

Proceedings of the 2014 International Workshop on Movement and Computing

View full text Add to dashboard Cite

While human-human or human-object interactions involve very rich, complex and nuanced gestures, gestures as they are captured for human-computer interaction remain relatively simplistic. Our approach is to consider the study of variation of motion input as a way of understanding expression and expressivity in human-computer interaction and in order to propose computational solutions for capturing and using these expressive variations. The paper reports an attempt at drawing the lines of design guidelines for modeling systems adapting to motion variations. We propose to illustrate them through two case studies: the first model is used to estimate temporal and geometrical motion variations while the second is used to track variations of motion dynamics. These case studies are illustrated in two applications.

show abstract

Online Gesture Analysis and Control of Audio Processing

Cited by 28 publications

References 36 publications

Mapping Through Listening

Mapping Through Listening

A multimodal probabilistic model for gesture--based control of sound synthesis

Motion Modeling for Expressive Interaction

Contact Info

Product

Resources

About