Transfer Learning for Piano Sustain-Pedal Detection

Abstract

Detecting piano pedalling techniques in polyphonic music remains a challenging task in music information retrieval. While other piano-related tasks, such as pitch estimation and onset detection, have seen improvement through applying deep learning methods, little work has been done to develop deep learning models to detect playing techniques. In this paper, we propose a transfer learning approach for the detection of sustainpedal techniques, which are commonly used by pianists to enrich the sound. In the source task, a convolutional neural network (CNN) is trained for learning spectral and temporal contexts when the sustain pedal is pressed using a large dataset generated by a physical modelling virtual instrument. The CNN is designed and experimented through exploiting the knowledge of piano acoustics and physics. This can achieve an accuracy score of 0.98 in the validation results. In the target task, the knowledge learned from the synthesised data can be transferred to detect the sustain pedal in acoustic piano recordings. A concatenated feature vector using the activations of the trained convolutional layers is extracted from the recordings and classified into frame-wise pedal press or release. We demonstrate the effectiveness of our method in acoustic piano recordings of Chopin's music. From the crossvalidation results, the proposed transfer learning method achieves an average F-measure of 0.89 and an overall performance of 0.84 obtained using the micro-averaged F-measure. These results outperform applying the pre-trained CNN model directly or the model with a fine-tuned last layer.

Music Genre Classification using Optimized Sequential Neural Network

“…In a work by B. Liang et al [4], transfer learning has been considered a better yet complex procedure to train various new models than machine learning algorithms. It's a knowledge-based approach that trains new models which may not have a proper training dataset based on the existing models by deriving logic between various attributes.…”

“…Earlier classifications of music genre were done by K. Simonyan and A. Zisserman [13] using CNN based on research by K. Choi et al [12] auto tagging methods of the audio files to train the model. B. Liang et al [4] had used a transfer learning approach which was based on the knowledge acquired from these pre-trained and auto-tagged CNN models which were further applied to the target task for music genre classification. The accuracy obtained was around 89% for the above approach.…”

“…The accuracy obtained was around 89% for the above approach. The transfer learning approach used in [4] enables the use for various classification and regression models, for instance we see that in the paper the features that were extracted and obtained for testing from the pre-trained CNNs, were actually trained using a large dataset in the source task. Thus, the obtained features were used in the target task for classification of the recordings which had proved to be a great advantage.…”

“…Thus, the obtained features were used in the target task for classification of the recordings which had proved to be a great advantage. But it can be observed in [4] that the use of transfer learning required them to create a very complex algorithm and the search for right models for creating a training dataset that shall be using knowledge-based training makes it way more complex.…”

NIME Scores: a Systematic Review of How Scores Have Shaped Performance Ecologies in NIME

“…Simultaneously, the system records a score without affecting the interaction nor the performance (Figure 8). For instance, Beson and colleagues created a sound spatialization system and a shaping interface that allows users to record scores of what they perform [81]; Liang and colleagues developed a system that analyzes piano pedaling and records a score of it [82]; finally, MuDI allows for real-time creation of scores for films [83].…”