Annotator subjectivity in harmony annotations of popular music

Abstract

Reference annotation datasets containing harmony annotations are at the core of a wide range of studies in music information retrieval (MIR) and related fields. The majority of these datasets contain single reference annotations describing the harmony of each piece. Nevertheless, studies showing differences among annotators in many other MIR tasks make the notion of a single 'ground-truth' reference annotation a tenuous one. In this paper, we introduce and analyse the Chordify Annotator Subjectivity Dataset (CASD) containing chord labels for 50 songs from 4 expert annotators in order to gain a better understanding of the differences between annotators in their chord label choice. Our analysis reveals that annotators use distinct chord-label vocabularies, with low chord-label overlap across all annotators. Between annotators, we find only 73 percent overlap on average for the traditional major-minor vocabulary and 54 percent overlap for the most complex chord labels. A factor analysis reveals the relative importance of triads, sevenths, inversions and other musical factors for each annotator on their choice of chord labels and reported difficulty of the songs. Our results further substantiate the existence of a harmonic 'subjectivity ceiling': an upper bound for evaluations in computational harmony research. Current state-of-the-art chord-estimation systems perform beyond this subjectivity ceiling by about 10 percent. This suggests that current ACE algorithms are powerful enough to tune themselves to particular annotators' idiosyncrasies. Overall, our results show that annotator subjectivity is an important factor in harmonic transcriptions, which should inform future studies into harmony perception and computational models of harmony.

Exploring annotations for musical pattern discovery gathered with digital annotation tools

“…For instance, Flexer and Grill (2016) discovered a rather low inter-annotator agreement for the MIREX music similarity task, unveiling the problem of using a single-reference annotation for evaluating similarity algorithms. Koops et al (2019) reached similar conclusions for the chord estimation task, showing low interannotator agreement for chord annotations between musical experts. Furthermore, Balke et al (2016) showed how the evaluation of automatic melody finding algorithms depends heavily on the choice of the human annotator for providing the ground truth.…”

On Local Keys, Modulations, and Tonicizations

“…There are datasets available with encoded roman numeral analysis annotations [14,34,46], which could be used for studying changes of key in the manner that we have presented here. However, it is important to acknowledge that roman numeral annotations are subject to issues such as ambiguity and disagreement [2,12,22,42], which may have implications for determining where the changes of key occur. For example, the dashed regions in Figure 1 show the areas where the key is ambiguous.…”

Improving Audio Chord Estimation by Alignment and Integration of Crowd-Sourced Symbolic Music

“…2 However, a study on MIREX results from 2010 to 2015 reveals a stagnation in ACE performance (Scholz et al, 2016). Besides, various studies (Ni et al, 2013;Humphrey and Bello 2015;Koops et al, 2019) throw light on another issue of ACE: chord annotations are inherently subjective, which can result in multiple, heterogeneous chord annotations. Recently, Koops et al (2019) introduced a 50-song data set of popular music, annotated by 4 professional musicians, and found only 73% overlap on average for the traditional major-minor vocabulary.…”

“…Besides, various studies (Ni et al, 2013;Humphrey and Bello 2015;Koops et al, 2019) throw light on another issue of ACE: chord annotations are inherently subjective, which can result in multiple, heterogeneous chord annotations. Recently, Koops et al (2019) introduced a 50-song data set of popular music, annotated by 4 professional musicians, and found only 73% overlap on average for the traditional major-minor vocabulary. The currently common practice to evaluate ACE by comparing the results to a single reference annotation is disputed by Humphrey and Bello (2015); Ni et al (2013) and Koops et al (2019).…”

“…Recently, Koops et al (2019) introduced a 50-song data set of popular music, annotated by 4 professional musicians, and found only 73% overlap on average for the traditional major-minor vocabulary. The currently common practice to evaluate ACE by comparing the results to a single reference annotation is disputed by Humphrey and Bello (2015); Ni et al (2013) and Koops et al (2019). Ni et al (2013) and Koops et al (2019) even claim that modern ACE systems have started to overfit on standard ACE data sets, effectively mimicking the specific aspects of MIREX's reference annotations.…”