Understanding Effects of Subjectivity in Measuring Chord Estimation Accuracy

Abstract

To assess the performance of an automatic chord estimation system, reference annotations are indispensable. However, owing to the complexity of music and the sometimes ambiguous harmonic structure of polyphonic music, chord annotations are inherently subjective, and as a result any derived accuracy estimates will be subjective as well. In this paper, we investigate the extent of the confounding effect of subjectivity in reference annotations. Our results show that this effect is important, and they affect different types of automatic chord estimation systems in different ways. Our results have implications for research on automatic chord estimation, but also on other fields that evaluate performance by comparing against human provided annotations that are confounded by subjectivity.

Automatic chord label personalization through deep learning of shared harmonic interval profiles

“…In an experiment we compare training of our chord label personalization system on multiple reference annotations with training on a commonly used single reference annotation. In the first case we train a dnn (dnn ship ) on ships derived from a dataset introduced by Ni et al [2013] containing 20 popular songs annotated by five annotators with varying degrees of musical proficiency. In the second case, we train a dnn (dnn iso ) on the hip of the Isophonics (iso) single reference annotation [Mauch et al, 2009].…”

“…Ignoring inversions, the complete dataset from Ni et al [2013] Table 2: Chord label personalization accuracies for the five annotators labels, comprised of five annotators using 87, 74, 62, 81 and 26 unique chord labels respectively. The intersection of the chord labels of all annotators contains just 21 chord labels meaning that each annotator uses a quite distinct vocabulary of chord labels.…”

“…On the other hand, it can also be argued that there is no single best reference annotation, and that chord labels are correct with varying degrees of "goodness-of-fit" depending on the target audience [Ni et al, 2013]. In particular for richly orchestrated, harmonically complex music, different chord labels can be chosen for a part, depending on the instrument, voicing or the annotators' chord label vocabulary.…”

Leveraging repetition for improved automatic lyric transcription in popular music

“…The techniques presented in this work may be of use in other MIR scenarios in which a consensus annotation from many candidates is desired, which has been the subject of at least two recently-published papers on beat tracking [30] and chord detection [31]. We would like to explore this in future work, as well as create high-quality acoustic and language models for ALR.…”

Evaluation and combination of pitch estimation methods for melody extraction in symphonic classical music

“…Inspired by the definitions of melody in (Poliner et al 2007; Selfridge-Field 1998), we collected excerpts in which human listeners agreed in their 'essence', that is, the sequence of notes that they hum or sing to represent it. The problem with interannotator agreement has been discussed in tasks such as chord recognition (Ni et al 2013) or music similarity (Flexer 2014). Several MIR datasets have also involved more than one annotator during their creation, e.g.…”