Prior-free Data Acquisition for Accurate Statistical Estimation

Abstract

We study a data analyst's problem of acquiring data from self-interested individuals to obtain an accurate estimation of some statistic of a population, subject to an expected budget constraint. Each data holder incurs a cost, which is unknown to the data analyst, to acquire and report his data. The cost can be arbitrarily correlated with the data. The data analyst has an expected budget that she can use to incentivize individuals to provide their data. The goal is to design a joint acquisition-estimation mechanism to optimize the performance of the produced estimator, without any prior information on the underlying distribution of cost and data. We investigate two types of estimations: unbiased point estimation and confidence interval estimation.Unbiased estimators: We design a truthful, individually rational, online mechanism to acquire data from individuals and output an unbiased estimator of the population mean when the data analyst has no prior information on the cost-data distribution and individuals arrive in a random order. The performance of this mechanism matches that of the optimal mechanism, which knows the true cost distribution, within a constant factor. The performance of an estimator is evaluated by its variance under the worst-case cost-data correlation. Confidence intervals:We characterize an approximately optimal (within a factor 2) mechanism for obtaining a confidence interval of the population mean when the data analyst knows the true cost distribution at the beginning. This mechanism is efficiently computable. We then design a truthful, individually rational, online algorithm that is only worse than the approximately optimal mechanism by a constant factor. The performance of an estimator is evaluated by its expected length under the worst-case cost-data correlation.

Binary Scoring Rules that Incentivize Precision

“…Our work also fits into the broad category of principal-agent problems. For example, works such as [3][4][5]11] consider a learning principal who incentivizes agents to make costly effort and produce an accurate data point. Again, the models are fairly distinct, as these works focus on more sophisticated learning problems (e.g.…”

“…(2) ′′′ is Riemann integrable on any closed subinterval of (0, 1). 5 (3) ∃ > 1 4 , and 0 > 0 s.th. for all ∈ (0, 0 ):…”

A Protocol for a Market of Machine Learning Models

“…The authors propose a method to convert a large class of no-regret algorithms into online postedprice and learning mechanisms. Some related topics are discussed in a large number of papers (see, for example, (Chen, Immorlica, Lucier, Syrgkanis, & Ziani, 2018;Chen & Zheng, 2019;Gast, Ioannidis, Loiseau, & Roussillon, 2020;Zhang, Arafa, Wei, & Berry, 2021)).…”

The Privacy Paradox and Optimal Bias-Variance Trade-offs in Data Acquisition

“…A growing line of work has focused on the design of such optimal data acquisition mechanisms, e.g., Roth and Schoenebeck [2012], , Chen and Zheng [2019], Abernethy et al [2015]. Initiated by Abernethy et al [2015], Roth and Schoenebeck [2012], this line of work has led to the design of mechanisms for unbiased estimation with minimal variance in a variety of settings.…”

“…However, in this literature it is assumed that all individuals will participate, thus unbiased estimation is possible. The trade-off between bias and variance has been ignored to this point with the exception of Chen and Zheng [2019], which still assumes all individuals will participate and does not consider privacy concerns. Further, this line of work has not considered the issues created by information leakage due to correlation between the participants.…”