ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data

Abstract

Most of the current supervised automatic music transcription (AMT) models lack the ability to generalize. This means that they have trouble transcribing real-world music recordings from diverse musical genres that are not presented in the labelled training data. In this paper, we propose a semi-supervised framework, ReconVAT, which solves this issue by leveraging the huge amount of available unlabelled music recordings. The proposed ReconVAT uses reconstruction loss and virtual adversarial training. When combined with existing U-net models for AMT, ReconVAT achieves competitive results on common benchmark datasets such as MAPS and MusicNet. For example, in the few-shot setting for the string part version of MusicNet, ReconVAT achieves F1-scores of 61.0% and 41.6% for the note-wise and note-with-offset-wise metrics respectively, which translates into an improvement of 22.2% and 62.5% compared to the supervised baseline model. Our proposed framework also demonstrates the potential of continual learning on new data, which could be useful in real-world applications whereby new data is constantly available. CCS CONCEPTS• Applied computing → Sound and music computing; • Computing methodologies → Semi-supervised learning settings; Neural networks.

Deep-Learning Architectures for Multi-Pitch Estimation: Towards Reliable Evaluation

“…Such score-audio pairs of classical music were considered to generate the MusicNet (MuN) dataset [18], an excellent resource of 330 license-free audio files with pre-synchronized pitch annotations that crucially stimulated the advancement of MPE [19]- [21]. 1 MuN is a very challenging dataset for MPE, yielding consistently and substantially worse results (F-measure below 75% [20]) across all studies compared to piano datasets such as MAPS or MAESTRO (up to 90% Fmeasure [7]).…”

“…As a major advancement, U-net models [35] were shown to improve performance of AMT [3], [20]- [23], [36] and other MIR tasks [37], [38]. More recently, the inclusion of self-attention components into U-nets [3], [21] and other models [39] was applied successfully to MPE. Most of our architectures rely on the U-net paradigm, enhanced with self-attention components as in [3] and other extensions.…”

“…Cheuk et al [36] consider two U-Nets, one for transcription and one for spectrogram reconstruction, connected with Bidirectional LSTM components and a reconstruction loss. Aiming towards better model generalization, they combined the reconstruction paradigm with virtual adversarial training [21]. To put a semantic meaning to the bottom part (bottleneck), some authors train the U-net in a multi-task fashion [22], [41] by extracting a voice activity estimate from the bottleneck, applied for monophonic pitch estimation of the predominant melody [41] or the bass line [22].…”

“…This is the case for the MusicNet dataset [18], a great resource of 330 license-free audio files with pre-synchronized pitch annotations. MusicNet crucially stimulated the advancement of MPE beyond the piano scenario [19]- [21] and has become the benchmark dataset for beyond-piano AMT. For this reason, we consider it as our central evaluation scenario.…”

“…Such aspects of robustness and generalization are particularly important for MPE systems based on supervised deep learning, which has become the de-facto standard approach [3], [5], [6], [19]. U-net models led to major advancements in MPE [20], [22], [23] and, recently, the inclusion of self-attention components [3], [21] or multi-task strategies [20], [22] into U-nets showed further success. In this paper, we take up such network architectures based on U-net and self-attention components [3] and propose several extensions.…”