Real-Time Audio-to-Score Alignment Using Particle Filter for Coplayer Music Robots

Abstract

Our goal is to develop a coplayer music robot capable of presenting a musical expression together with humans. Although many instrument-performing robots exist, they may have difficulty playing with human performers due to the lack of the synchronization function. The robot has to follow differences in humans' performance such as temporal fluctuations to play with human performers. We classify synchronization and musical expression into two levels: (1) melody level and (2) rhythm level to cope with erroneous synchronizations. The idea is as follows: When the synchronization with the melody is reliable, respond to the pitch the robot hears, when the synchronization is uncertain, try to follow the rhythm of the music. Our method estimates the score position for the melody level and the tempo for the rhythm level. The reliability of the score position estimation is extracted from the probability distribution of the score position. The experimental results demonstrate that our method outperforms the existing score following system in 16 songs out of 20 polyphonic songs. The error in the prediction of the score position is reduced by 69% on average. The results also revealed that the switching mechanism alleviates the error in the estimation of the score position.

A Stochastic Temporal Model of Polyphonic MIDI Performance with Ornaments

“…When dealing with temporal information, tempo must be estimated simultaneously with score positions. One can consider a two-dimensional state space of score position and tempo and perform a joint estimation [28,24,15]. Generally, inferences using Monte Carlo methods or discretization methods are then necessary since the search space is large.…”

Bayesian Audio-to-Score Alignment Based on Joint Inference of Timbre, Volume, Tempo, and Note Onset Timings

“…Modeling the sequence of features also has two major approaches: dynamic time warping (DTW) and hidden Markov models (HMMs). Also, a few approaches have been proposed that use conditional random fields (Joder et al 2010) or continuous statespace models (Duan and Pardo 2011b;Montecchio and Cont 2011;Otsuka et al 2011). Dynamic time warping takes as input two feature sequences, along with a measure of dissimilarity between any two features.…”

Automatic music accompaniment allowing errors and arbitrary repeats and jumps

“…Although a joint model of score position and tempo can in general be used to simultaneously estimate both, such a model typically necessitates large computational cost (see, for example, Ref. [8]). To avoid a significant increase of this cost, we alternately perform tempo estimation and score-position estimation.…”