Real-Time Audio-to-Score Alignment of Music Performances Containing Errors and Arbitrary Repeats and Skips

Abstract

Abstract-This paper discusses real-time alignment of audio signals of music performance to the corresponding score (a.k.a. score following) which can handle tempo changes, errors and arbitrary repeats and/or skips (repeats/skips) in performances. This type of score following is particularly useful in automatic accompaniment for practices and rehearsals, where errors and repeats/skips are often made. Simple extensions of the algorithms previously proposed in the literature are not applicable in these situations for scores of practical length due to the problem of large computational complexity. To cope with this problem, we present two hidden Markov models of monophonic performance with errors and arbitrary repeats/skips, and derive efficient score-following algorithms with an assumption that the prior probability distributions of score positions before and after repeats/skips are independent from each other. We confirmed real-time operation of the algorithms with music scores of practical length (around 10000 notes) on a modern laptop and their tracking ability to the input performance within 0.7 s on average after repeats/skips in clarinet performance data. Further improvements and extension for polyphonic signals are also discussed.

A Novel Interface for the Graphical Analysis of Music Practice Behaviors

“…Such automatic methods have struggled with practice scenarios that may contain many errors, pauses, and repetitions. Recent research by Nakamura et al ( 2016 ) and Sagayama et al ( 2014 ) has aimed to tackle the problem of alignment of performances with incorrect notes. An alignment system trained especially on practice data containing tempo changes, restarts, and incorrect notes could be integrated into the MPB framework in the future.…”

Structure-Aware Audio-to-Score Alignment Using Progressively Dilated Convolutional Neural Networks

“…Apart from JumpDTW and NWTW, which focus on offline alignment, work on online score following [13] has demonstrated the effectiveness of HMMs for modeling variations from the score. While this method focuses on real-time score following for monophonic music, our work deals with offline audio-to-score alignment for polyphonic music performance.…”

“…Another work related to ours is that proposed by Jiang et al [14], which focuses on offline score alignment for the practice scenario. Similar to Nakamura et al [13], their approach is also based on HMMs; but they propose using pitch trees and beam search to model skips. However, their method struggles with pieces containing both backward and forward jumps, which is an important challenge we tackle using our progressively dilated convolutional models.…”

Handling Structural Mismatches in Real-time Opera Tracking

“…In [8], Arzt and Widmer propose a two-level hypothesis tracking process, where a higher-level tracker evaluates alternative possible positions in the score and feeds the best candidates to a lower-level tracker that applies standard OLTW. Finally, in [13], Nakamura et al use a HMM variant that connects all score events together with additional repeat and skip probabilities.…”