When Experts Disagree: Response Aggregation and its Consequences in Expert Surveys

Abstract

Political scientists use expert surveys to assess the latent features of political actors. Experts, though, are unlikely to be equally informed and assess all actors equally well. The literature acknowledges variance in measurement quality but pays little attention to the implications of uncertainty for aggregating responses. We discuss the nature of the measurement problem in expert surveys. We then propose methods to assess the ability of experts to judge where actors stand and to aggregate expert responses. We examine the effects of aggregation for a prominent survey in the literature on party politics and EU integration. Using a Monte Carlo simulation, we demonstrate that it is better to aggregate expert responses using the median or modal response, rather than the mean.

How and How Much Does Expert Error Matter? Implications for Quantitative Peace Research

“…Because latent concepts are difficult to observe, and Likert-scales provide only rough guidance for translating continuous concepts to a categorical scale, experts will inevitably disagree when coding. 4 However, disagreement may also result from variation in expert scale perception (differential item functioning, of DIF) and reliability (Clinton & Lewis 2008, Bakker et al 2014, Lindstädt, Proksch & Slapin 2018, Marquardt & Pemstein 2018b). In the case of concepts like identity-based discrimination, such variation may be particularly problematic.…”

“…In this article, I focus on three main methods: 1) the normalized average; 2) the median; and 3) latent variable models, here a modified Bayesian ordinal item response theory (IRT) model. A primary virtue of the first two methods is that they are straightforward, with Lindstädt, Proksch & Slapin (2018) arguing that the median is a robust alternative to the more commonly-used average. The virtue of the third modeling strategy is that it can account for both DIF and variation in expert reliability; indeed, recent research has illustrated that an IRT model outperforms both the normalized average and the median in recovering latent values when the level of expert error is high (Marquardt & Pemstein 2018a, Marquardt & Pemstein 2018b.…”

“…However, expert-coded data come with potential disadvantages. Experts are susceptible to different sources of error (Clinton & Lewis 2008, Bakker, Jolly, Polk & Poole 2014, Marquardt & Pemstein 2018b; such error may bias results in statistical analyses (Lindstädt, Proksch & Slapin 2018). These concerns are particularly acute in the context of quantitative peace research.…”

Estimating Latent Traits from Expert Surveys: An Analysis of Sensitivity to Data Generating Process

“…Lindstädt, Proksch and Slapin (2018) motivate the BMed approach by postulating an expert rating process that is largely analagous to the A-M model. However, while both A-M and IRT models assume an unbounded interval-scale latent space, Lindstädt, Proksch and Slapin (2018) assume that latent values fall on the interval (l, u) ∈ R. In contrast to A-M models they also assume that expert intercept, slope and variance parameters are uniformly, not normally, distributed.…”

“…Recent work has suggested three alternative methods for accounting for these forms of error: the bootstrapped median (BMed) and two types of latent variable models, Aldrich-McKelvey (A-M) scaling (Bakker, Jolly, Polk and Poole, 2014;Aldrich and McKelvey, 1977) and item-response theory (IRT) models (Clinton and Lewis, 2008;Pemstein et al, 2018). The BMed approach is both simple and arguably more robust than MpSD (Lindstädt, Proksch and Slapin, 2018), while both latent variable modeling approaches rely on rather complicated techniques to adjust for DIF and random error.…”