Co-Performing Music with AI: Real-Time Performance Control Using Speech and Gestures

Abstract

We present an interactive music AI system that enables users to co-create expressive performances of notated music using speech and gestures. The system provides multi-modal interactive dialog-based control of performance rendering via smartphones and is accessible to people regardless of their musical background. We train a deep learning music performance rendering model on sheet music and associated performances with notated performance directions and user-system interaction data. Users have the opportunity to actively participate in the performance process. A speech- and gesture-based feedback loop with interactive learning improve the accuracy of performance rendering control. We believe that many people can express aspects of music performance using natural human expressions such as speech, voice, and gestures, and that by hearing the music follow their communicated intent, they can achieve deeper immersion and enjoyment of music than otherwise possible. With this work we pursue the goal of developing novel, fulfilling, and accessible music making experiences for large numbers of people who are not currently musically active.