Evaluation methods for information retrieval systems come in three types: offline evaluation, using static data sets annotated for relevance by human judges; user studies, usually conducted in a labbased setting; and online evaluation, using implicit signals such as clicks from actual users. For the latter, preferences between rankers are typically inferred from implicit signals via interleaved comparison methods, which combine a pair of rankings and display the result to the user. We propose a new approach to online evaluation called multileaved comparisons that is useful in the prevalent case where designers are interested in the relative performance of more than two rankers. Rather than combining only a pair of rankings, multileaved comparisons combine an arbitrary number of rankings. The resulting user clicks then give feedback about how all these rankings compare to each other. We propose two specific multileaved comparison methods. The first, called team draft multileave, is an extension of team draft interleave. The second, called optimized multileave, is an extension of optimized interleave and is designed to handle cases where a large number of rankers must be multileaved. We present experimental results that demonstrate that both team draft multileave and optimized multileave can accurately determine all pairwise preferences among a set of rankers using far less data than the interleaving methods that they extend.

No abstract

It is far from obvious to find logical counterparts to crytographic protocol primitives. In logic, a common assumption is that agents are perfectly rational and have no computational limitations. This creates a dilemma. If one merely abstracts from computational aspects, protocols become trivial and the difference between tractable and intractable computation, surely an essential feature of protocols, disappears. This way, the protocol gets lost. On the other hand, if one 'merely' (scare quotes indeed) models agents with computational limitations (or otherwise bounded rationality), very obvious aspects of reasoning become problematic. That way, the logic gets lost. We present a novel way out of this dilemma. We propose an abstract logical architecture wherein public and private, or symmetric keys, and their roles in crytographic protocols, all have formal counterparts. Instead of having encryption and decryption done by a principal, the agent sending or receiving messages, we introduce additional, virtual, agents to model that, so that one-way-function aspects of computation can be modelled as constraints on the communication between principals and these virtual counterparts. In this modelling it does not affect essential protocol features if agents are computationally unlimited. We have implemented the proposal in a dynamic epistemic model checker called DEMO.

In this paper we investigate Kripke models, used to model knowledge or belief in a static situation, and action models, used to model communicative actions that change this knowledge or belief. The appropriate notion for structural equivalence between modal structures such as Kripke models is bisimulation: Kripke models that are bisimilar are modally equivalent. We would like to find a structural relation that can play the same role for the action models that play a prominent role in information updating. Two action models are equivalent if they yield the same results when updating Kripke models. More precisely, two action models are equivalent if it holds for all Kripke models that the result of updating with one action model is bisimilar to the result of updating with the other action model. We propose a new notion of action emulation that characterizes the structural equivalence of the important class of canonical action models. Since every action model has an equivalent canonical action model, this gives a method to decide the equivalence of any pair of action models. We also give a partial result that holds for the class of all action models. Our results extend the work in [4].