Tempo Driven Audio-to-Score Alignment Using Spectral Decomposition and Online Dynamic Time Warping

Abstract

In this article, we present an online score following framework designed to deal with automatic accompaniment. The proposed framework is based on spectral factorization and online Dynamic Time Warping (DTW) and has two separated stages: preprocessing and alignment. In the first one, we convert the score into a reference audio signal using a MIDI synthesizer software and we analyze the provided information in order to obtain the spectral patterns (i.e., basis functions) associated to each score unit. In this work, a score unit represents the occurrence of concurrent or isolated notes in the score. These spectral patterns are learned from the synthetic MIDI signal using a method based on Non-negative Matrix Factorization (NMF) with Beta-divergence, where the gains are initialized as the ground-truth transcription inferred from the MIDI. On the second stage, a non-iterative signal decomposition method with fixed spectral patterns per score unit is used over the magnitude spectrogram of the input signal resulting in a distortion matrix that can be interpreted as the cost of the matching for each score unit at each frame. Finally, the relation between the performance and the musical score times is obtained using a strategy based on online DTW, where the optimal path is biased by the speed of interpretation. Our system has been evaluated and compared to other systems, yielding reliable results and performance.

Online score‐informed source separation in polyphonic mixtures using instrument spectral patterns

“…In the comparison, four configurations of our method are presented. GT denotes the variant of our proposed method in which the perfect annotation score is used to guide the factorization stage, ScoreFree denotes the variant in which no alignment stage is used, and, therefore, the gains matrix A p , j ( t ) is initialized with random values, and Offline and Online make reference to variants whose alignment stages have been implemented following the Offline and Online approaches described in the work of Rodriguez‐Serrano et al Furthermore, we are also going to contrast our proposal with Oracle version, which uses the individual sources to establish the best separation that can be obtained using the proposed softmask reconstruction strategy. It sets an upper bound of all the configurations of the proposed method.…”

“…Finally, at the DTW block , the optimum path across matrix D ( τ , t ) is computed using DTW algorithm to provide the alignment between the score and performance times. More information about the alignment described earlier can be found in the works of Carabias‐Orti et al and Rodriguez‐Serrano et al…”

“…Finally, at the DTW block, the optimum path across matrix D( , t) is computed using DTW algorithm to provide the alignment between the score and performance times. More information about the alignment described earlier can be found in the works of Carabias-Orti et al 17 and Rodriguez-Serrano et al 18 The software ReMAS (Real-time Musical Accompaniment System), designed to track the reproduction of a musical piece with the aim to match the score position into its symbolic representation on a digital sheet, was presented in the work of Alonso et al 24 ReMAS shows that it is possible to exploit efficiently several cores of an ARM ® processor, or a GPU accelerator, reducing the processing time per frame in a few milliseconds in most of the cases. On the other hand, a parallel online DTW solution based on a client-server architecture implemented for multicore architectures (86, 64, and ARM ® ) is proposed in another work of Alonso et al 25 Both softwares have a high degree of similarity to the alignment stage.…”

Online/offline score informed music signal decomposition: application to minus one

“…In fact, classical offline systems rely on cost measures between events in the score and in the performance. Two methods well known in speech recognition have been extensively used in the literature: statistical approaches (e.g., hidden Markov model (HMM)) [22][23][24][25][26] and dynamic time warping (DTW) [19,[27][28][29].…”

“…Although several online audio-to-score approaches have been developed in the literature [22,28,29], only the works in [6,22,30] combined score alignment with SS in an online fashion. In [22] and in its extension [6], the alignment is performed using a hidden Markov process model, where each audio frame is associated with a 2-D state of score position and tempo.…”