Nonmanipulable Bayesian testing

Abstract

This paper considers the problem of testing an expert who makes probabilistic forecasts about the outcomes of a stochastic process. I show that, as long as uninformed experts do not learn the correct forecasts too quickly, a likelihood test can distinguish informed from uninformed experts with high prior probability. The test rejects informed experts on some data-generating processes; however, the set of such processes is topologically small. These results contrast sharply with many negative results in the literature.

Calibration and Expert Testing

“…Stewart (2011) formalizes this idea as follows. One may wonder whether such a prior will help a tester to separate informed from ignorant experts.…”

Claim Validation

“…A growing literature studies whether strategic experts can avoid rejection (see, among several contributions, Al-Najjar and Weinstein (2008), Al-Najjar, Smorodinsky, Sandroni and Weinstein (2010), Babaio , Blumrosen, Lambert and Reingold (2011), Cesa-Bianchi and Lugosi (2006), Chassang (2013), Dekel and Feinberg (2006), Feinberg and Stewart (2008), Feinberg and Lambert (2014), Fortnow and Vohra (2009), Foster and Vohra (1998), Fudenberg and Levine (1999), Gradwohl and Salant (2011), Gradwohl and Shmaya (2013), Hu and Shmaya (2013), Lehrer (2001), Olszewski and Peski (2011), Olszewski andSandroni (2008,2009a-b), Sandroni (2003), Shmaya (2008), Stewart (2011), and Vovk and Shafer (2005)). For a review, see Foster and Vohra (2013) and Olszewski (2011).…”

Testable forecasts

“…In particular, there exist non-convex paradigms that are not testable and convex paradigms that are testable. 4 Likelihood-ratio tests play an important role in Stewart (2011). Stewart proposes a framework where the tester is a Bayesian endowed with a prior over laws and the forecaster is evaluated according to a likelihood-ratio test against the predictions induced by the prior.…”