Extracting Wisdom from Experts and Small Crowds: Strategies for Improving Informant-based Measures of Political Concepts

Abstract

Social scientists have increasingly turned to expert judgments to generate data for difficult-to-measure concepts, but getting access to and response from highly expert informants can be costly and challenging. We examine how informant selection and post-survey response aggregation influence the validity and reliability of measures built from informant observations. We draw upon three surveys with parallel survey questions of candidate characteristics to examine the trade-off between expanding the size of the local informant pool and the pool's level of expertise. We find that a “wisdom-of-crowds” effect trumps the benefits associated with the expertise of individual informants when the size of the rater pool is modestly increased. We demonstrate that the benefits of expertise are best realized by prescreening potential informants for expertise rather than post-survey weighting by expertise.

What Makes Experts Reliable?

“…Higher scores indicate that the expert diverges less from other experts than experts with lower scores, which we interpret as proxying her reliability in the more colloquial sense. 2 Reliability is not necessarily the same as accuracy (Maestas, Buttice and Stone, 2014). However, assessing accuracy directly is an impossible task in this dataset, given that there is no concrete reference point for coding accuracy of latent variables.…”

What makes experts reliable? Expert reliability and the estimation of latent traits

“…Analyses of reliability correlates also provide tentative evidence regarding the characteristics of more reliable experts, which may facilitate decisions on expert recruitment and retention. Although research stresses that expertise is important for data validity (Maestas et al, 2014), potential correlates of intra-expert variation in this context remain largely unexplored.…”

Estimating Candidates’ Political Orientation in a Polarized Congress

“…Clinton et al (2004) show that the results of this model are almost identical to the results of Nominate. We downloaded the replication data from the Dataverse (Maestas et al 2013). We use the inclc_pc09 variable for incumbent placements, dlc_pc10 for Democratic candidates' placements, and rlc_pc10 for Republican candidates' placements.…”

“…In this paper, we examine the properties of six recent measures of candidates' political orientations in different domains. 1 Specifically, we examine the properties of measures based on candidates' National Political Awareness Test (NPAT) responses from Project Vote Smart (Ansolabehere et al 2001;Montagnes and Rogowski 2014), their state legislative voting records (Shor and McCarty 2011), their Twitter followers (Barberá 2015), their campaign donor networks (Bonica 2013b(Bonica , 2014, the perceptions of survey respondents (Aldrich and McKelvey 1977;Hare et al 2015;Ramey 2016), and expert assessments (Stone and Simas 2010;Joesten and Stone 2014;Maestas et al 2014).…”