A Test of Consensus in Aesthetic Evaluation among Professional Critics of Modern Music

Abstract

To evaluate the approximate level of aesthetic consensus among a large sample of professionals, modern music critics were sampled from known music rating books that rated comparable genres of music, including the ratings of 352 critics and 5161 albums of randomly chosen musicians. All critic pairs who had rated at least 30 albums in common were analyzed (N = 139 pairs). Overall, 87.0% of critic pairs showed significant positive correlations in their album ratings, and another 2.9% showed marginally significant positive correlations (average r = .49). Not a single significant negative correlation occurred, and nonsignificant correlations occurred in only 10.1% of the critic pairs. Overall, reasonable aesthetic consensus appears to exist among most modern music critics, even without specific, agreed-upon rating scales, methods or assumptions, and despite preliminary evidence that not all professional critics are equally good at what they do. The superiority of correlation measures over agreement measures of aesthetic consensus is also discussed.

Degrees of Quality: A Method for Quantifying Aesthetic Impact

“…It is the same for aesthetic ratings, we do not necessarily know why people are agreeing, yet we still can know when they are agreeing. In support of this view, without any predetermined definitions or rating methods, a large sample of modern music critic pairs have been shown to be moderately consensual in their independent ratings of albums, with 86% of critic pairs showing positive correlations and not a single pair showing a negative correlation (average r = + 0.49; Lundy, 2010a). It would, of course, be nice to know the underlying factors that create consensus, but this does not appear to be essential.…”

“…At least two exemplars need to be used for each level to ensure that it is not the specific characteristics of one particular artwork that is being represented, but instead their shared level of perceived aesthetic impact (e.g., for pop/rock music, the author ended up using John Lennon's God and Brian Eno's Golden Hours to represent level 100). These songs are top tier exemplars that the author personally used, and whether others would agree would be an empirical question, although both of these songs do come from consistently highly rated albums by professional critics (Lundy, 2010a). Moreover, across the songs that were used as exemplars, the average correlation between ratings and the ratings of Rolling Stone critics' ratings from the corresponding albums was + 0.87 (DeCurtis, Henke, & George-Warren, 1992; Marsh & Swenson, 1979, 1983.…”

“…Other music rating studies had already successfully used 13-point (Heyduk, 1975) and 14-point (Krugman, 1943) rating scales. Moreover, in the author's study of modern music critics, moderate consensus was obtained while many of the critics used 11-point scales (Lundy, 2010a). Payne (1997) suggested that depending on the sophistication level of the raters, scales having as many as 20 categories could be used (DLAIRM starts with 21).…”

“…Relevant and supportive data so far have come from five sources. First, to demonstrate general soundness in the author's overall music evaluation tendencies while using DLAIRM (i.e., aesthetic judgment), he compared his ratings of a random sample of 364 albums with a database of professional critic ratings that he had compiled for another study (Lundy, 2010a). Album ratings showed significant positive correlations with six out of nine critics with no negative correlations (average r = + 0.33, range = + 0.11 to + 0.63).…”

“…Yet others have argued that only certain aspects can be consensual, such as whether or not the intended artistic purpose was achieved, but that taste is purely subjective (Kaufman, 2002). Despite of the philosophical debate, empirical research has demonstrated that even without agreeing on any particular method of evaluation, reasonable consensus has been established in varied areas of aesthetics, including music (Farnsworth, 1949(Farnsworth, , 1950Lundy, 2010a;North & Hargreaves, 1995, 1996, film (Boor, 1990(Boor, , 1992Simonton, 2004) and visual art (Child, 1962a;1962b).…”