A Two-Stage Approach to Note-Level Transcription of a Specific Piano

Abstract

This paper presents a two-stage transcription framework for a specific piano, which combines deep learning and spectrogram factorization techniques. In the first stage, two convolutional neural networks (CNNs) are adopted to recognize the notes of the piano preliminarily, and note verification for the specific individual is conducted in the second stage. The note recognition stage is independent of piano individual, in which one CNN is used to detect onsets and another is used to estimate the probabilities of pitches at each detected onset. Hence, candidate pitches at candidate onsets are obtained in the first stage. During the note verification, templates for the specific piano are generated to model the attack of note per pitch. Then, the spectrogram of the segment around candidate onset is factorized using attack templates of candidate pitches. In this way, not only the pitches are picked up by note activations, but the onsets are revised. Experiments show that CNN outperforms other types of neural networks in both onset detection and pitch estimation, and the combination of two CNNs yields better performance than a single CNN in note recognition. We also observe that note verification further improves the performance of transcription. In the transcription of a specific piano, the proposed system achieves 82% on note-wise F-measure, which outperforms the state-of-the-art.

The Impact of Audio Input Representations on Neural Network based Music Transcription

“…Various models such as support vector machines (SVM) [5,6], restricted Boltzmann machines (RBM) [4], long-short term memory neural networks [7], and convolutional neural networks (CNN) [8,9] have been developed to tackle this task. For example, Wang et al [10] integrate non-negative matrix factorization (NMF) with a CNN in order to improve transcription accuracy. Hawthorne et al [11] split the AMT into three sub-tasks: onset detection, frame activation, and velocity estimation, which allows them to achieve state-of-the art transcription accuracy on piano music.…”

Polyphonic Piano Transcription Using Autoregressive Multi-State Note Model

“…This onset-aware model significantly reduced note-level false positive errors, which is critical in perceptual evaluation of the transcription. Similar multi-state note modeling approaches are found in [4,[7][8][9][10] and some detect even more phases of note envelope including onset, sustain and offset [11,12]. As such, various versions of note state representations have been suggested so far and showed improved performances.…”

“…Most of recent approaches in polyphonic piano transcription are based on deep learning. The model architectures are diverse, including CNN [2,3,10,12], RNN [9,15], CRNN [4,5,16], and U-Net [17]. The loss function is typically the cross-entropy between predicted and ground truth labels but also includes the adversarial loss [5].…”

“…The loss function is typically the cross-entropy between predicted and ground truth labels but also includes the adversarial loss [5]. An important direction in designing a neural network architecture is detecting note onset explicitly apart from the binary on/off states [4,9,10,12], considering that piano sound starts with a percussive tone but, after the attack park, it slowly decays with a harmonic tone [18]. This multi-state note modeling even including note offset was already explored before the DNN approaches become dominant [7,8,11].…”

Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms

“…Comprehensive experimental comparison of DNN-and NMF-based ADT methods have been reported in [3]. Convolutional neural networks (CNNs), for example, have been used for extracting local time-frequency features from an input spectrogram [11][12][13][14][15]. Recurrent neural networks (RNNs) are expected to learn the temporal dynamics inherent in music and have successfully been used, often in combination with CNNs, for estimating the smooth onset probabilities of drum sounds at the frame level [16][17][18].…”