Optimal adaptive testing: Informativeness and incentives

Abstract

We introduce a learning framework in which a principal seeks to determine the ability of a strategic agent. The principal assigns a test consisting of a finite sequence of tasks. The test is adaptive: each task that is assigned can depend on the agent's past performance. The probability of success on a task is jointly determined by the agent's privately known ability and an unobserved effort level that he chooses to maximize the probability of passing the test. We identify a simple monotonicity condition under which the principal always employs the most (statistically) informative task in the optimal adaptive test. Conversely, whenever the condition is violated, we show that there are cases in which the principal strictly prefers to use less informative tasks. several anonymous referees, and various seminar and conference participants for helpful comments and suggestions. Rya Sciban and Young Wu provided excellent research assistance. We are very grateful for the financial support provided by the Social Sciences and Humanities Research Council of Canada.1 Learning a worker's ability by observing their performance on differentially informative tasks is a problem that goes back to at least Prescott and Visscher (1980). 2 This assumption is consistent with the observation in Baker et al. (1988) that in many organizations, promotion is the only means used for providing incentives. 3 This feature of the optimal test is reminiscent of the optimal contract in the (two outcome version of the) dynamic, pure moral hazard framework of Hölmstrom and Milgrom (1987), where the agent's overall compensation depends only on the number-and not the order-of successes. 4 Prasad (2009) and Ferreira and Sah (2012) are recent examples of models where workers can be either generalists or specialists.

Strategic Communication With Minimal Verification

“…This might not be desirable, either because any single question reveals too noisy a signal, or because the students may not know their relative skills perfectly. Here, the natural solution is to offer several problems instead of one, which in turn poses the problem of the optimal number of questions an exam should have, and how to choose their topics for each student in an optimal way (see also Deb and Stewart (), who studied a similar question with a one‐dimensional type space). We hope that the insights from this paper will help address these and other questions.…”

“…Dziuda and Salas () studied a cheap‐talk model in which the receiver may learn that the sender lied, but without learning what the truth was; in their model, discovery of lies is random and exogenous, unlike ours where verification is the object of design. Deb and Stewart () studied an adaptive testing problem where there is a limit on the number of tests that may be performed, as in our model. There is also a growing branch of the mechanism design literature with costly verification, started by Townsend (), and more recently including Kartik and Tercieux (), Ben‐Porath, Dekel, and Lipman (), and Erlanson and Kleiner ().…”

Probabilistic Verification in Mechanism Design

“…3 We consider nonbinary tests in Section 8; our main results go through. 4 The same procedure is used in Deb and Stewart (2018). They allow the principal to conduct tests (termed tasks) sequentially before making a binary classification.…”

Information transmission in persuasion models with imperfect verification

“…A similar assumption called group monotonicity is made inDeb and Stewart (2018), where the authors study how a firm should determine the tasks its workers perform so as to best determine their ability. Despite the similarity of the two conditions, group monotonicity is neither implied nor implies that p is ordered.13 If the agent was able to observe s prior to reporting, then the DM could not gain from communicating with the agent unless he was able to commit to a reporting strategy.…”

“…1212 A similar assumption called group monotonicity is made in Deb and Stewart (2018), where the authors study how a firm should determine the tasks its workers perform so as to best determine their ability. Despite the similarity of the two conditions, group monotonicity is neither implied nor implies that p is ordered. …”