Eliciting properties of probability distributions

Abstract

We investigate the problem of truthfully eliciting an expert's assessment of a property of a probability distribution, where a property is any real-valued function of the distribution such as mean or variance. We show that not all properties are elicitable; for example, the mean is elicitable and the variance is not. For those that are elicitable, we provide a representation theorem characterizing all payment (or "score") functions that induce truthful revelation. We also consider the elicitation of sets of properties. We then observe that properties can always be inferred from sets of elicitable properties. This naturally suggests the concept of elicitation complexity; the elicitation complexity of property is the minimal size of such a set implying the property. Finally we discuss applications to prediction markets.

Self-financed wagering mechanisms for forecasting

“…For instance, the property Γ corresponding to the probability of an event A is Γ(P ) = P (A), that of the median of a random variable X is Γ(P ) = sup m {m : P (X < m) < 1/2}. As in Lambert et al [11], we say that a report r (respectively, a probability P ) is admissible when r (respectively Γ(P )) falls in the interior of Γ's range, which is an open interval for continuous properties. We are interested in wagering mechanisms wherein players report sets or vectors of property values.…”

“…Distribution properties, introduced in Lambert et al [11], are a convenient way to model information on probability distributions. A distribution property Γ(P ) is defined as a function that assigns a real value to any probability distribution P in a given convex domain.…”

“…Similar to scoring rules [16], Lambert et al [11] define reward functions that truthfully elicit single properties or sets of properties from a risk neutral agent as follows.…”

“…We assume Γ is continuous and not locally constant, 4 and denote by (a, b) the interval of admissible reports. As in Lambert et al [11], we define a distribution property to be elicitable when there exists a strictly proper score function for that property. We say that a function is smooth when it is twice continuously differentiable.…”

“…Now, Theorem 2 of Lambert et al [11] shows that elicitable properties may be expressed as linear constraints. In particular, it shows that there exist random variables Yi and Yj such that EP [Yi] = 0 ⇔ Γ(P ) = r * i and EP [Yj] = 0 ⇔ Γ(P ) = r * j .…”