Computational Models of Expressive Music Performance: A Comprehensive and Critical Review

Abstract

Expressive performance is an indispensable part of music making. When playing a piece, expert performers shape various parameters (tempo, timing, dynamics, intonation, articulation, etc.) in ways that are not prescribed by the notated score, in this way producing an expressive rendition that brings out dramatic, affective, and emotional qualities that may engage and affect the listeners. Given the central importance of this skill for many kinds of music, expressive performance has become an important research topic for disciplines like musicology, music psychology, etc. This paper focuses on a specific thread of research: work on computational music performance models. Computational models are attempts at codifying hypotheses about expressive performance in terms of mathematical formulas or computer programs, so that they can be evaluated in systematic and quantitative ways. Such models can serve at least two purposes: they permit us to systematically study certain hypotheses regarding performance; and they can be used as tools to generate automated or semi-automated performances, in artistic or educational contexts. The present article presents an up-to-date overview of the state of the art in this domain. We explore recent trends in the field, such as a strong focus on data-driven (machine learning) approaches; a growing interest in interactive expressive systems, such as conductor simulators and automatic accompaniment systems; and an increased interest in exploring cognitively plausible features and models. We provide an in-depth discussion of several important design choices in such computer models, and discuss a crucial (and still largely unsolved) problem that is hindering systematic progress: the question of how to evaluate such models in scientifically and musically meaningful ways. From all this, we finally derive some research directions that should be pursued with priority, in order to advance the field and our understanding of expressive music performance.

An Interdisciplinary Review of Music Performance Analysis

“…While most of the studies mentioned above make use of statistical methods to extract, summarize, visualize, and investigate patterns in the performance data, researchers have also investigated modeling approaches to better understand performances. Several overview articles exist covering research on generating expressive musical performance (Cancino-Chacón et al, 2018;Kirke and Miranda, 2013;Widmer and Goebl, 2004). In this subsection, however, we primarily focus on methods that model performance parameters leading to useful insights, while ignoring the generative aspect.…”

On the Characterization of Expressive Performance in Classical Music: First Results of the Con Espressione Game

“…At the intersection of research on music emotion and computational models of performance is the study of the relation between performance parameters (timing, dynamics) and emotion [4][5][6][7][8]. For an overview of computational modeling of music performance, we refer the reader to [14].…”

Towards Explaining Expressive Qualities in Piano Recordings: Transfer of Explanatory Features Via Acoustic Domain Adaptation

“…For instance, the DCASE 2019 and 2020 challenges had dedicated tasks on Acoustic Scene Classification with multiple/mismatched recording devices. 1 The machine learning answer to this problem is research on effective methods for transfer learning and (supervised and unsupervised) domain adaptation.…”

“…In a large project, 2 we aim at studying the elusive concept of expressivity in music with computational and, specifically, machine learning methods. One aspect of that is the art of expressive performance, the subtle, continuous shaping of musical parameters such as tempo, timing, dynamics, and articulation by experienced musicians, while playing a piece, in this way imbuing the piece with particular expressive and emotional qualities [1]. The Con Espressione Game was a large-scale data collection effort we set up in order to obtain personal descriptions of perceived expressive qualities, with the goal of studying human perception and characterisation of expressive aspects in performances of the same pieces by different artists [2].…”