Seven Years of INEX Interactive Retrieval Experiments – Lessons and Challenges

Abstract

Abstract. This paper summarizes a major effort in interactive search investigation, the INEX i-track, a collective effort run over a seven-year period. We present the experimental conditions, report some of the findings of the participating groups, and examine the challenges posed by this kind of collective experimental effort.

A Manifesto on Resource Re-Use in Interactive Information Retrieval

“…This necessitates sophisticated research designs with rich combinations of data collection and analysis methods and a research infrastructure with many interdependent IR system components. In the past few decades, there have been several multi-year IIR campaigns that have attempted to re-use the experimental setup from year to year within the campaign, such as the TREC Interactive Track (1997)(1998)(1999)(2000)(2001)(2002) [39], the INEX Interactive Track (2004-2010) [38,47,56], the Cultural Heritage in CLEF (CHiC) Interactive Task (2013) [46,58], and the interactive Social Book Search (iSBS) task (2014)(2015)(2016) [19,20,24].…”

A Standardised Format for Exchanging User Study Instruments

“…In the TREC Interactive Track (1997)(1998)(1999)(2000)(2001)(2002) [6], participants used similar protocols, but tested different variables and used partly inconsistent measurements. The INEX Interactive Track (2004-2010 [10] employed standardised tasks and data collection protocols, demonstrating that shared tasks across research institutions are possible in IIR as well. Nevertheless, the overlap between consecutive years was not as high as desired due to changing corpora, IR systems, and participants.…”

Report on the Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2018)

“…In his keynote on "The importance and challenges for standardization in IIR Evaluation -Basis for an iRepository", Nils Pharo discussed the INEX interactive track [7], which ran from 2004-2010. The interactive track at INEX studied user interactions based on tasks, which were loosely associated with the system-centered INEX tracks in order to compare user and system perspectives.…”

“…One way is to make materials available via a single repository. Another is to encourage researchers to share their materials via open data sharing platforms such as Dataverse 6 or the Open Science Framework 7 . Where components of a study are related to previous studies (e.g., different editions of the same interactive track), it would be useful to describe the differences between editions/versions, as well as reasons for any changes with respect to these previous studies.…”