Fusing different information retrieval systems according to query-topics: a study based on correlation in information retrieval systems and TREC topics

Abstract

Abstract.To evaluate Information Retrieval Systems on their effectiveness, evaluation programs such as TREC offer a rigorous methodology as well as benchmark collections. Whatever the evaluation collection used, effectiveness is generally considered globally, averaging the results over a set of information needs. As a result, the variability of system performance is hidden as the similarities and differences from one system to another are averaged. Moreover, the topics on which a given system succeeds or fails are left unknown. In this paper we propose an approach based on data analysis methods (correspondence analysis and clustering) to discover correlations between systems and to find trends in topic/system correlations. We show that it is possible to cluster topics and systems according to system performance on these topics, some system clusters being better on some topics. Finally, we propose a new method to consider complementary systems as based on their performances which can be applied for example in the case of repeated queries. We consider the system profile based on the similarity of the set of TREC topics on which systems achieve similar levels of performance. We show that this method is effective when using the TREC ad hoc collection.

Acquiring Subject Participation for Information Security Survey Research: A Content and Correspondence Analysis Approach

“…A perceptual map produced by CA is capable of providing "a better understanding and certainly more easily present relations from a picture than from a large table of coefficients", (SPSS 1998, p. 1-1). For example, CA has been used in information systems research to study strategic information systems planning (Remenyi 1992), website characteristics (Jowkar and Didegah 2010), and information retrieval systems (Bigot et al 2011). …”

Evaluation of information retrieval systems using structural equation modeling

“…Approaches inspired to data mining to understanding retrieval failures were also proposed by Bigot et al [2011]. Reliability analysis also investigated the best practices for learning to rank deployments by Macdonald et al [2013].…”

Evaluating the Interest of Revamping Past Search Results

“…A framework dedicated to improve effectiveness measures such as Average Precision (AP), where a set of documents is assigned to the best system cluster (i.e. best answer given a query) can be found in [3]. Several approaches in IR use past queries for query expansion.…”