Game Theory for Data Science: Eliciting Truthful Information

Faltings, Boi; Radanovic, Goran; Brachman, Ronald J.; Stone, Peter

doi:10.2200/s00788ed1v01y201707aim035

Cited by 33 publications

(27 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We provide a brief review of the related work in peer prediction and suggest Faltings and Radanovic [2017] for a detailed discussion. We focus our discussion on related work about minimal mechanisms but remark that we are not aware of any non-minimal mechanisms (following from the work of Prelec [2004]) that handle agent heterogeneity.…”

Section: Related Workmentioning

confidence: 99%

Peer Prediction with Heterogeneous Users

Agarwal

Mandal

Parkes

et al. 2020

ACM Trans. Econ. Comput.

View full text Add to dashboard Cite

Peer prediction mechanisms incentivize agents to truthfully report their signals, in the absence of a verification mechanism, by comparing their reports with those of their peers. Prior work in this area is essentially restricted to the case of homogeneous agents, whose signal distributions are identical. This is limiting in many domains, where we would expect agents to differ in taste, judgment, and reliability. Although the Correlated Agreement (CA) mechanism [Shnayder et al. 2016a] can be extended to handle heterogeneous agents, there is a new challenge of efficiently estimating agent signal types. We solve this problem by clustering agents based on their reporting behavior, proposing a mechanism that works with clusters of agents, and designing algorithms that learn such a clustering. In this way, we also connect peer prediction with the Dawid and Skene [1979] literature on latent types. We retain the robustness against coordinated misreports of the CA mechanism, achieving an approximate incentive guarantee of ε -informed truthfulness. We show on real data that this incentive approximation is reasonable in practice, even with a small number of clusters.

show abstract

Section: Related Workmentioning

confidence: 99%

Peer Prediction with Heterogeneous Users

Agarwal

Mandal

Parkes

et al. 2020

ACM Trans. Econ. Comput.

View full text Add to dashboard Cite

show abstract

“…In Fig. 2, we present a taxonomy of information eliciting mechanisms used for single value and probability distribution values elicitation (see [22]). In Fig.…”

Section: Related Workmentioning

confidence: 99%

Eliciting Truthful Data From Crowdsourced Wireless Monitoring Modules in Cloud Managed Networks

et al. 2020

View full text Add to dashboard Cite

To facilitate efficient cloud managed resource allocation solutions, collection of key wireless metrics from multiple access points (APs) at different locations within a given area is required. In unlicensed shared spectrum bands collection of metric data can be a challenging task for a cloud manager as independent self-interested APs can operate in these bands in the same area. We propose to design an intelligent crowdsourcing solution that incentivizes independent APs to truthfully measure/report data relating to their wireless channel utilization (CU). Our work focuses on challenging scenarios where independent APs can take advantage of recurring patterns in CU data by utilizing distribution aware strategies to obtain higher reward payments. We design truthful reporting methods that utilize logarithmic and quadratic scoring rules for reward payments to the APs. We show that when measurement computation costs are considered then under certain scenarios these scoring rules no longer ensure incentive compatibility. To address this, we present a novel reward function which incorporates a distribution aware penalty cost that charges APs for distorting reports based on recurring patterns. Along with synthetic data, we also use real CU data values crowdsourced using multiple independent measuring/reporting devices deployed by us in the University of Oulu.

show abstract

“…These include: (a) Garcin and Faltings (2014) showing that peer prediction can elicit forecasts which accuracies are comparable to those elicited via prediction markets 3 ; (b) Gao et al (2014) showing that collusion can occur if a small group of users is repeatedly asked to report their private information; (c) Radanovic, Faltings, and Jurca (2016) showing that robust peer prediction designs perform well in peer grading. More general overview of the literature on incentive mechanism design can be found in Faltings and Radanovic (2017).…”

Section: Related Workmentioning

confidence: 99%

The Effectiveness of Peer Prediction in Long-Term Forecasting

Mandal

Radanovic

David

2020

AAAI

View full text Add to dashboard Cite

In human forecasting, proper scoring rules are used to elicit effort in providing accurate probability forecasts of future events. A challenge, though, is that users do not receive feedback about their forecasts until the outcomes are realized. Nor is it clear whether these schemes are effective in motivating continual attention, and updating forecasts on difficult or dynamically changing problems, for which there is a continuous inflow of new information over time.Through a large-scale experiment on Amazon Mechanical Turk (MTurk), we investigate whether peer prediction methods can be used to complement methods of proper scoring rules, and improve engagement of users and ultimately the quality of forecasts. Peer prediction provides immediate feedback, by comparing one forecaster's prediction with that of another, this feedback provided as rank placement or through incentive payments. One of a very small number of experimental studies into peer prediction, ours is the first to test peer prediction in this hybrid role.We show that providing daily feedback through peer prediction has a significant effect in increasing engagement with the forecasting platform. Moreover, a hybrid scheme that combines scoring rules with peer prediction feedback (via rank feedback) is, together with the basic scoring rule method, generally the best for accuracy. Since the hybrid scheme also improves user engagement, this suggests that the hybrid scheme would provide the best accuracy for longer term forecasting events.

show abstract

Game Theory for Data Science: Eliciting Truthful Information

Cited by 33 publications

References 39 publications

Peer Prediction with Heterogeneous Users

Peer Prediction with Heterogeneous Users

Eliciting Truthful Data From Crowdsourced Wireless Monitoring Modules in Cloud Managed Networks

The Effectiveness of Peer Prediction in Long-Term Forecasting

Contact Info

Product

Resources

About