A technical framework for automatic perceptual evaluation of singing quality

Abstract

Human experts evaluate singing quality based on many perceptual parameters such as intonation, rhythm, and vibrato, with reference to music theory. We proposed previously the Perceptual Evaluation of Singing Quality (PESnQ) framework that incorporated acoustic features related to these perceptual parameters in combination with the cognitive modeling concept of the telecommunication standard Perceptual Evaluation of Speech Quality to evaluate singing quality. In this study, we present further the study of the PESnQ framework to approximate the human judgments. First, we find that a linear combination of the individual perceptual parameter human scores can predict their overall singing quality judgment. This provides us with a human parametric judgment equation. Next, the prediction of the individual perceptual parameter scores from the PESnQ acoustic features show a high correlation with the respective human scores, which means more meaningful feedback to learners. Finally, we compare the performance of early fusion and late fusion of the acoustic features in predicting the overall human scores. We find that the late fusion method is superior to that of the early fusion method. This work underlines the importance of modeling human perception in automatic singing quality assessment.

Deep Learning Approaches in Topics of Singing Information Processing

“…1) Intonation Accuracy: Intonation accuracy or pitch accuracy evaluation has been the most common method for singing quality assessment, primarily because studies have shown that intonation accuracy is one of the most important perceptual parameters when music experts assess singing quality [42], [43]. In one of the earliest studies, Lal [44] proposed a pitchbased similarity measure to compare a test singing clip to the reference singing clip.…”

“…Although MIDI notes used as the reference approximately represent the sequence of sung notes, they are unable to repre-sent human voice since singing voice comprises of pitch transitions, modulations, and different voice timbres. Therefore, studies for singing quality assessment have explored comparison with an ideal reference singing rendition [46], [47], [43], instead of the MIDI notes of the song. The drawback of this approach is that the choice of an ideal singing rendition is subjective.…”

“…3) Perceptual Quality: According to music psychology studies of human perception, humans first convert the perceived singing audio into a weighted representation of the identified perceptual parameters [50], and then make a judgment of overall quality in a holistic manner [43], [29], [28]. For example, Nakano et al [12] fuse features derived from intonation and vibrato to classify recordings of professional singers as either 'good' or 'bad' and achieve an accuracy of up to 87%, depending on the gender of the singer, though it is reference-independent.…”

“…certain phrases sung with bad pitch or bad rhythm) have a greater subjective impact than distributed error. Furthermore, Gupta et al [43] explored early and late fusion of pitch, rhythm, timbre, and vibrato related features to map those objective features to the overall singing quality judgment scores by human music experts. They found that the late fusion method achieved a higher correlation with human judgment than early fusion.…”

“…Lyrics transcription in solo singing [209], [211], singer identification and query by singing [232], singing style and intonation pattern analysis [60], [233] DSing (DAMP Sing! Lyrics Curated) [209] 150 hours curated English songs data from the DAMP dataset; removed noisy data https://github.com/groadabike/ Kaldi-Dsing-task Lyrics transcription in solo singing [209], [211], [215] DAMP-VSEP [234] 11,494 compositions (155 countries, 36 languages, 6456 artists) with backing tracks, one or more isolated vocals, and a mixture of the two https://zenodo.org/record/3553059 Singing voice separation [173] DAMP Aligned [203] 50 hours training data, 2.3 hours test; lyrics aligned and short segments https://github.com/chitralekha18/ lyrics-aligned-solo-singing-dataset Lyrics transcription in solo singing [203], [226], [224] DALI [228] 134 hours English polyphonic song utterances with aligned lyrics https://github.com/gabolsgabs/DALI Lyrics transcription in polyphonic music NUS48E [97] 2.8 hours recordings of the sung and spoken lyrics of 48 (20 unique) English songs by 12 subjects and transcriptions and duration annotations at the phonelevel https://smcnus.comp.nus.edu.sg/ nus-48e-sung-and-spoken-lyrics-corpus/ Speech-singing conversion [235], singing synthesis, pronunciation evaluation [236], phoneme alignment in solo singing NHSS [126] 100 songs sung and spoken by 10 singers, resulting in total of 7 hours audio data https://hltnus.github.io/NHSSDatabase/ index.html Speech-singing conversion, singing synthesis, lyrics alignment in solo singing NUS48E+ SingEval [43] 2 songs, 20 singers; music experts labels on pitch, rhythm, etc. https://github.com/chitralekha18/ PESnQ APSIPA2017 Singing skill evaluation [47], [43], [22], [59] DAMP SingEval [39] 400 renditions (4 songs, 100 singers per song), each rated by humans on the basis of singing quality https://github.com/chitralekha18/ SingEval.git Singing skill evaluation [39], [58], [57],…”