Rate the Raters: A Note on Wine Judge Consistency

Abstract

Much literature shows that the ratings assigned by wine judges are uncertain, some authors have proposed that judges be tested, and a few wine competitions do test judges. However, no literature or competition has yet proposed a test or rating for judges based on realistic competition conditions. This article uses coefficients of multiple correlation to rate each of 54 judges who assigned ratings to 2,811 wines entered in a commercial competition. Results show that there is a strong and positive correlation between the ratings assigned by most judges to most wines. However, those correlations also show that the ratings assigned by approximately 10% of judges are indistinguishable from random assignments. Using correlations to rate the raters, a program is underway to monitor those judges and variations in competition protocol that may affect their ratings. (JEL Classifications: A10, C00, C10, C12, D12)

Judging reliability at wine and water competitions

“…We find that the inter-rater reliability of water judging at the Taste Awards is generally better than random chance, similar to wine judging at some competitions (including, in particular, the 1976 Judgment of Paris), and worse than wine judging at other competitions (including, in particular, a recent year of the California State Fair's commercial wine competition studied by Bodington, 2020). The fact that the Taste Awards have a lower inter-rater reliability than at least some wine competitions may be due to water experts having blind-tasting abilities that are less developed and aesthetic standards that are less established than wine experts.…”

“…Previous studies on the inter-rater reliability of wine judging One measure of the inter-rater reliability between two judges' judgments is the Pearson product-moment correlation coefficient (Olkin et al, 2015). That measure, which is appropriate for cardinal data, has been used for several analyses of wine judging (e.g., Ashton, 2012;Bodington, 2020). A coefficient of (plus or minus) unity means there is a perfect (direct or inverse) linear association between the judges' judgments.…”

“…Although a Pearson correlation coefficient can be used to compare the judgments of only two judges at a time, Ashton (2012), Bodington (2020), and others have compared the judgments of more than two judges by calculating a Pearson correlation coefficient for each possible pair of judges (i.e., pairing each judge with every other judge, except themselves) and then taking a simple average of all those correlation coefficients. Hereafter, we will refer to that as the "average pairwise correlation."…”

Wine Ratings: Seeking a Consensus among Tasters via Normalization, Approval, and Aggregation

“…We illustrated our protocols using the Judgement of Paris and 2018 en-primeur Bordeaux wines, rated by five international experts. 18 Our analysis concludes that the way in which we build the consensuses is quite different from the usual simple aggregations of ratings or rankings. We nevertheless believe that it is important to notice that different (plausible) decisions to build a consensual ranking generate different outcomes.…”

“…Numerical wine ratings are very popular. The scientific literature has paid attention to aspects such as (1) inconsistency of ratings in blind tastings (Lindley, 2006;Hodgson, 2008;Bodington, 2017Bodington, , 2020, (2) consensus among experienced wine experts (Ashton, 2012(Ashton, , 2013Cao, 2014;Luxen, 2018), (3) variations in the severity of experts (Masset, Weisskopf, and Cossutta, 2015;Stuen, Miller, and Stone, 2015), and (4) consumers' demand for wine ratings (Ashenfelter and Jones, 2013;Marks, 2015Marks, , 2020. In this article, we aimed at providing a comprehensive framework to reach a consensus among tasters' opinions (expressed via wine ratings).…”