The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019)

Self Cite

When researchers speak of BM25, it is not entirely clear which variant they mean, since many tweaks to Robertson et al.'s original formulation have been proposed. When practitioners speak of BM25, they most likely refer to the implementation in the Lucene open-source search library. Does this ambiguity "matter"? We attempt to answer this question with a large-scale reproducibility study of BM25, considering eight variants. Experiments on three newswire collections show that there are no significant effectiveness differences between them, including Lucene's often maligned approximation of document length. As an added benefit, our empirical approach takes advantage of databases for rapid IR prototyping, which validates both the feasibility and methodological advantages claimed in previous work.

Section: Introductionmentioning

confidence: 99%

Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants

Kamphuis

Vries

Boytsov

et al. 2020

Self Cite

“…Although each artifact broke in its own idiosyncratic way, we do notice common themes. Given continued interest in reproducibility, most notably a new Docker-based iteration of OSIRRC in 2019 [3], 4 these "lessons learned" might form the basis of future best practices.…”

Section: Lessons Learnedmentioning

confidence: 99%

“…PRIMAD implicitly assumes a static view of reproducibility, as opposed to the processoriented view we advocate. More concretely, from the successes as well as failures in replicating OSIRRC 2015, we are able to extract a number of "lessons learned" that can be further distilled into best practices, especially to inform ongoing efforts such as the latest iteration of OSIRRC [3].…”

Section: Introductionmentioning

confidence: 99%

Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments

Lin

Zhang

2020

Self Cite

This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time "achievement". As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those computational artifacts still functional? Perhaps not surprisingly, we are not able to replicate most of the retrieval runs encapsulated by those artifacts in a modern computational environment. We outline the various idiosyncratic reasons why, distilled into a series of "lessons learned" to help form an emerging set of best practices for the long-term sustainability of reproducibility efforts.

“…Workshops deal with the reproducibility either re-or proactively. For example, the CENTRE workshop [9] challenges participants to reconstruct IR systems and their results, whereas The Open-Source IR Replicability Challenge (OSIRRC) [7] motivated participants to package their retrieval systems and corresponding software dependencies in advance to prepare them for appropriate reuse.…”

Section: Related Workmentioning

confidence: 99%

“…The results of our experimental setups showed that we can replicate the outcomes fairly well, whereas reproduced outcomes are significantly lower. Having the reimplementation of an ad-hoc retrieval system at hand, we decided to contribute it to the OSIRRC@SIGIR2019 workshop [7]. All contributions resulted in an image library of Docker images to which we contributed the IRC-CENTRE2019 image [4].…”

Section: Preliminary Work and Research Proposalmentioning

confidence: 99%

Reproducible Online Search Experiments

Breuer

2020

In the empirical sciences, the evidence is commonly manifested by experimental results. However, very often, these findings are not reproducible, hindering scientific progress. Innovations in the field of information retrieval (IR) are mainly driven by experimental results as well. While there are several attempts to assure the reproducibility of offline experiments with standardized test collections, reproducible outcomes of online experiments remain an open issue. This research project will be concerned with the reproducibility of online experiments, including real-world user feedback. In contrast to previous living lab attempts by the IR community, this project has a stronger focus on making IR systems and corresponding results reproducible. The project aims to provide insights concerning key components that affect reproducibility in online search experiments. Outcomes help to improve the design of reproducible IR online experiments in the future.