Assessment of Student Music Performances Using Deep Neural Networks

Abstract

Music performance assessment is a highly subjective task often relying on experts to gauge both the technical and aesthetic aspects of the performance from the audio signal. This article explores the task of building computational models for music performance assessment, i.e., analyzing an audio recording of a performance and rating it along several criteria such as musicality, note accuracy, etc. Much of the earlier work in this area has been centered around using hand-crafted features intended to capture relevant aspects of a performance. However, such features are based on our limited understanding of music perception and may not be optimal. In this article, we propose using Deep Neural Networks (DNNs) for the task and compare their performance against a baseline model using standard and hand-crafted features. We show that, using input representations at different levels of abstraction, DNNs can outperform the baseline models across all assessment criteria. In addition, we use model analysis techniques to further explain the model predictions in an attempt to gain useful insights into the assessment process. The results demonstrate the potential of using supervised feature learning techniques to better characterize music performances.

An Interdisciplinary Review of Music Performance Analysis

“…Recent methods, however, have transitioned towards using advanced machine learning techniques such as sparse coding (Han and Lee, 2014;Wu and Lerch, 2018c,a) and deep learning (Pati et al, 2018). Contrary to earlier methods which focused on hand-designing musically important features, these techniques input raw data (usually in the form of pitch contours or spectrograms) and train the models to automatically learn meaningful features so as to accurately predict the assessment ratings.…”

“…Tools for performance assessment evaluate one or more performance parameters typically related to the accuracy of the performance in terms of pitch and timing (Wu et al, 2016;Vidwans et al, 2017;Pati et al, 2018;Luo, 2015), or quality of sound (timbre) (Knight et al, 2011;Romani Picas et al, 2015;Narang and Rao, 2017). In building an assessment tool, the choice of parameters may depend on the proficiency level of the performer being assessed.…”

Applying Deep Learning Techniques to Estimate Patterns of Musical Gesture

“…To do so, the authors utilized the Myo sensor as an interactive input, reading the electromyogram signals of the performer's forearm, to trigger the sound manipulations. Pati et al (2018) proposed a hybrid model based on Mel spectrogram analysis from audio recordings of traditional music performance to pass the multidimensional data stream into a convolutional 1D layer projected to a recurrent neural network. The model receives the name of M-CRNN.…”

SonoSpace

“…Nakano et al [14] also manually extract features to assess the human singing voice based on the accuracy of pitch and rhythm and the presence of vibrato. Kumar et al [16] propose a system for recognizing pitch and rhythm, using convolutional neural networks. Many other systems have also been developed to manually extract the feature quantities of the performance and evaluate the quality of the performance [1,2,10,12,20,23].…”