Abstract.We consider the problem of acquiring relevance judgements for information retrieval (IR) test collections through crowdsourcing when no true relevance labels are available. We collect multiple, possibly noisy relevance labels per document from workers of unknown labelling accuracy. We use these labels to infer the document relevance based on two methods. The first method is the commonly used majority voting (MV) which determines the document relevance based on the label that received the most votes, treating all the workers equally. The second is a probabilistic model that concurrently estimates the document relevance and the workers accuracy using expectation maximization (EM). We run simulations and conduct experiments with crowdsourced relevance labels from the INEX 2010 Book Search track to investigate the accuracy and robustness of the relevance assessments to the noisy labels. We observe the effect of the derived relevance judgments on the ranking of the search systems. Our experimental results show that the EM method outperforms the MV method in the accuracy of relevance assessments and IR systems ranking. The performance improvements are especially noticeable when the number of labels per document is small and the labels are of varied quality.

We explore the phase diagram of the strongly correlated Hubbard model with intrinsic spin orbit coupling on the honeycomb lattice. We obtain the low energy effective model describing the spin degree of freedom. We study the resulting model within the Schwinger boson and Schwinger fermion approaches. The Schwinger boson approach gives the boundary between the spin liquid phase and the magnetically ordered phases, Neel order and incommensurate Neel order. We find that increasing the strength of the spin orbit coupling, narrows the width of the spin liquid region. The Schwinger fermion approach sheds further light on the nature of the spin liquid phase. We obtain three different candidates for the spin liquid phase within the mean field approximation which are gapless spin liquid, topological Mott insulator, and the chiral spin liquid phases. We argue that the gauge fluctuations and the instanton effect may suppress the first two spin liquids, while the chiral spin liquid is stable against gauge fluctuations due to its nontrivial topology.

We propose a mathematical framework for query selection as a mechanism for reducing the cost of constructing information retrieval test collections. In particular, our mathematical formulation explicitly models the uncertainty in the retrieval effectiveness metrics that is introduced by the absence of relevance judgments. Since the optimization problem is computationally intractable, we devise an adaptive query selection algorithm, referred to as Adaptive, that provides an approximate solution. Adaptive selects queries iteratively and assumes that no relevance judgments are available for the query under consideration. Once a query is selected, the associated relevance assessments are acquired and then used to aid the selection of subsequent queries. We demonstrate the effectiveness of the algorithm on two TREC test collections as well as a test collection of an online search engine with 1000 queries. Our experimental results show that the queries chosen by Adaptive produce reliable performance ranking of systems. The ranking is better correlated with the actual systems ranking than the rankings produced by queries that were selected using the considered baseline methods.

No abstract

No abstract