Streamlining Evaluation with ir-measures

Abstract

The material cannot be used for any other purpose without further permission of the publisher and is for private use only. There may be differences between this version and the published version. You are advised to consult the publisher's version if you wish to cite from it.

A Python Interface to PISA!

“…https://github.com/terrierteam/pyterrier_doc2query8 When we modified the PISA source to accept a file path (rather than using stdin) we found the indexing speed to be comparable to our approach. We acknowledge that this unexpected behaviour may be platform-dependent; we only tested on a system running Ubuntu 20.04.…”

“…Through a variety of tracing, we found that this was overhead sustained through the stdin piping mechanism it uses for reading input. 8 We Table 1: Indexing performance on MS MARCO (passage, v1) and TREC Robust04. Indexing duration was measured on the same machine under identical conditions, with the data provided in a format favourable to each system.…”

“…A retrieval transformer can be invoked over an entire set of queries to perform batch retrieval, using multi-threading to parallelise the retrieval: At an even higher level, researchers often want to measure the overall performance of retrieval models using standard benchmarks and measures. Here, we show how our Python PISA interface can be evaluated using PyTerrier's Experiment API, which calculates system performance using the ir_measures package [8].…”

“…We next examine how well our interface scales to multiple threads during retrieval. We use our interface to retrieve the top 100-1000 documents for each query using BM25 with default parameters, and vary the number of threads used (1,2,4,8,16). The test is conducted on a machine with 16 cores.…”

The Istella22 Dataset

“…Evaluation Metrics. We evaluate the performance of the methods using four well-known IR metrics, i.e., Precision (P), NDCG, MRR and MAP made available by the ir_measures library [52]. We evaluate precision at three different cut-offs, i.e., 𝑘 ∈ {1, 5, 10}.…”