Simplified Data Wrangling with ir_datasets

Abstract

Managing the data for Information Retrieval (IR) experiments can be challenging. Dataset documentation is scattered across the Internet and once one obtains a copy of the data, there are numerous different data formats to work with. Even basic formats can have subtle dataset-specific nuances that need to be considered for proper use. To help mitigate these challenges, we introduce a new robust and lightweight tool (ir_datasets) for acquiring, managing, and performing typical operations over datasets used in IR. We primarily focus on textual datasets used for ad-hoc search. This tool provides both a Python and command line interface to numerous IR datasets and benchmarks. To our knowledge, this is the most extensive tool of its kind. Integrations with popular IR indexing and experimentation toolkits demonstrate the tool's utility. We also provide documentation of these datasets through the ir_datasets catalog: https://ir-datasets.com/. The catalog acts as a hub for information on datasets used in IR, providing core information about what data each benchmark provides as well as links to more detailed information. We welcome community contributions and intend to continue to maintain and grow this tool.

C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

“…Finally, we ask the question: what if we can afford to translate MS MARCO so that we can use a translate-train model? To investigate, we utilize the Chinese translation of the MSMARCO-v1 training triples from ColBERT-X [32], which can also be accessed via ir_datasets [30] with the dataset key neumarco/zh 1 . Figure 2 shows that without C3, the ColBERT model improves from 0.352 to 0.421, which is still worse than zero-shot transfer models trained with C3 for CLIR, suggesting allocating effort to C3 rather than training a translation model when computational resources are limited.…”

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

“…For TREC graded relevance (0 = non relevant to 3 = perfect), we use the recommended binarization point of 2 for the recall metric. For out of domain experiments we refer to the ir_datasets catalogue [37] for collection specific information, as we utilized the standardized test sets for the collections.…”

“…Methodology. We selected seven datasets from the ir_datasets catalogue [37]: Bio medical (TREC Covid [50,52], TripClick [40], NFCorpus [4]), Entity centric (DBPedia Entity [14]), informal language (Antique [13], TREC Podcast [23]), news cables (TREC Robust 04 [49]). The datasets are not based on web collections, have at least 50 queries, and importantly contain judgements from both relevant and non-relevant categories.…”

DiffIR: Exploring Differences in Ranking Models' Behavior

“…It can be used as a command line tool and as a Python package that can be integrated with other tools. DiffIR is model-agnostic; in its most basic setting, it simply accepts TREC-formatted run files and an ir_datasets [32] dataset identifier to generate an HTML output. Metrics are calculated using pytrec_eval [39] via the ir_measures 3 package.…”

“…DiffIR can be run locally using the command: In the above command, run_1 and run_2 are files that contain the document rankings for each query and uses the standard TREC run format. The user must specify a dataset name supported by ir_datasets [32]. In the sample command above, DiffIR would select the top ten queries whose mean average precision varies the most between the two run files and renders the content as HTML.…”