Eliciting Expertise without Verification

Abstract

A central question 1 of crowdsourcing is how to elicit expertise from agents. This is even more difficult when answers cannot be directly verified. A key challenge is that sophisticated agents may strategically withhold effort or information when they believe their payoff will be based upon comparison with other agents whose reports will likely omit this information due to lack of effort or expertise.Our work defines a natural model for this setting based on the assumption that more sophisticated agents know the beliefs of less sophisticated agents.We then provide a mechanism design framework for this setting. From this framework, we design several novel mechanisms, for both the single and multiple tasks settings, that (1) encourage agents to invest effort and provide their information honestly; (2) output a correct "hierarchy" of the information when agents are rational. arXiv:1802.08312v2 [cs.GT] 22 May 2018 2 Mechanism Design ToolsWe use two key information theory ingredients in designing information elicitation mechanisms. The first ingredient is f -mutual information M I f (X; Y ) which measures the amount of information crossing two random variables X, Y . For example, if X is independent with Y -no information crosses X and Y , M I f (X; Y ) = 0. The second ingredient is proper scoring rule P S(x, p) which measures the accuracy of the prediction p even we only have one sample x of the outcome X.Both two ingredients have the information monotonicity property. If the information is measured by f -mutual information, any "data processing" on either of the random variables will decrease the amount of information crossing them. If the accuracy of a forecast is measured by a proper scoring rule, more information implies a more accurate forecast. f -mutual informationwhere f (⋅) is a convex function and f (1) = 0. Two commonly used f -divergences are KL divergence and total variation distance. Now we start to introduce f -mutual information.Given two random variables X, Y , let U X,Y and V X,Y be two probability measures where U X,Y is the joint distribution of (X, Y ) and V is the product of the marginal distributions of X and Y . Formally, for every pair of (x, y),If U X,Y is very different with V X,Y , the mutual information between X and Y should be high since knowing X changes the belief for Y a lot. If U X,Y equals to V X,Y , the mutual information between X and Y should be zero since X is independent with Y . Intuitively, the "distance" between U X,Y and V X,Y represents the mutual information between them.Definition 2.1 (f -mutual information [20]). The f -mutual information between X and Y is defined aswhere D f is f -divergence. Definition 2.2 (Conditional f -mutual information [20]). Given three random variables X, Y, Z, we define M I f (X; Y Z) as z P r[Z = z]M I f (X; Y Z = z) where M I f (X; Y Z = z) ∶= M I f (X ′ ; Y ′ ) where P r[X ′ = x, Y ′ = y] = P r[X = x, Y = y Z = z].

Auctions and Prediction Markets for Scientific Peer Review

“…Cai et al (2015) propose a continuous effort model for data elicitation, where exerting effort gives agents a lower variance signal. However, in our setting, lower effort reviews may not only be noisier, they may also be systematically biased (Kong and Schoenebeck, 2018). Gao et al (2016) caution that in practice (and as highly relevant in our case), agents have a variety of low-effort signals such as paper length, topic, etc., potentially leading to low-effort coordination into uninformative equilibria.…”

“…The IEWV literature is focused on the problem of eliciting truthful private 'signals' when the mechanism designer is unable to verify the responses; this may be because the ground truth is difficult or impossible for the mechanism to access or there is simply no ground truth since the responses are subjective. The key application domains of interest include tasks like crowdsourcing data, product ratings, community sensing, and peer grading (Cai et al, 2015;Radanovic et al, 2016;Dasgupta and Ghosh, 2013;Kong and Schoenebeck, 2018). More recent work is also interested in ensuring that private signals are obtained with effort.…”

“…With these considerations, Kong and Schoenebeck (2018) propose a Hierarchical Mutual Information Paradigm (HMIP) that constructs a hierarchy of criteria that capture a range of cheap to expensive signals, and pays agents the information gained from more expensive signals. This work builds on the Mutual Information Paradigm (Kong and Schoenebeck, 2019) which shows that mechanisms incentivize truth-telling when the payment is based on the mutual information between agents' responses, and this is what peer prediction and BTS are attempting to do.…”

“…Thus, our two main considerations are incentivizing reviewers to exert effort, and to report their true reviews. We achieve both by adapting the Target Differential Peer Prediction (TDPP) (Schoenebeck and Yu, 2020) to operate within the Hierarchical Mutual Information Paradigm (HMIP) (Kong and Schoenebeck, 2018). The Differential Peer Prediction mechanism is a recently proposed prediction market-style mechanism applicable to our small-group, heterogeneous reviewer setting and determines payments for reviews of each paper independently.…”

“…Other Assumptions We now list some other assumptions we make. First, we adopt a conditional independence assumption from Kong and Schoenebeck (2018), which states that a reviewer k 's private score on some criterion t contains all the information needed to predict their peer k's score on that criterion, given that peer's scores on lower level criteria. We write I(•) to represent the mutual information (see Appendix C for more details).…”