Axiomatic Retrieval Experimentation with ir_axioms

Bondarenko, Alexander; Fröbe, Maik; Reimer, Jan Heinrich; Stein, Benno; Völske, Michael; Hagen, Matthias

doi:10.1145/3477495.3531743

Cited by 3 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We finally supplement our findings with a theoretical analysis using information retrieval axioms [9] on the Touché 2020 data (Section 4.5) and find that all neural models violate the document length normalization axiom LNC2 [18], which is well-supported by BM25. Overall, our contributions are the following:…”

Section: Modelsupporting

confidence: 56%

“…DocT5query Expansion. We reuse the DocT5query [45] model 9 from BEIR [61] to expand documents in Touché 2020 with generated queries. We focus solely on generating queries using the premise (body) and not the conclusion (title) for all 382,545 documents in Touché 2020.…”

Section: Improving Effectiveness At Inference Timementioning

confidence: 99%

“…A higher agreement indicates that a retrieval model fulfills the theoretical constraint introduced in the axiom. These axioms can highlight the problems in neural models, and fixing these problems can improve the model's effectiveness [9], even when there is no strong correlation between axioms and relevance judgments [11]. While retrieval axioms can increase the effectiveness of neural retrieval models (e.g., when used for regularization [54]), dedicated axioms for neural retrieval models are still missing [64].…”

Section: Axiomatic Error Analysis On Touché 2020mentioning

confidence: 99%

“…Setup and Background. We conduct our axiomatic analysis using the ir_axioms framework [9]. 12 Because most axioms require theoretical preconditions that are rarely met in real-world datasets (e.g., requiring document pairs retrieved for the same query of identical length) [9], we first use synthetic document pairs derived from real documents and subsequently use real document pairs with the default length relaxation from ir_axioms.…”

Section: Axiomatic Error Analysis On Touché 2020mentioning

confidence: 99%

“…We conduct our axiomatic analysis using the ir_axioms framework [9]. 12 Because most axioms require theoretical preconditions that are rarely met in real-world datasets (e.g., requiring document pairs retrieved for the same query of identical length) [9], we first use synthetic document pairs derived from real documents and subsequently use real document pairs with the default length relaxation from ir_axioms. Given more than 20 previously proposed retrieval axioms [9], we include a subset of all axioms related to document length, term frequency, and semantic similarity in our analysis.…”

Section: Axiomatic Error Analysis On Touché 2020mentioning

confidence: 99%

See 4 more Smart Citations

Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR

Thakur,

Bonifacio,

Fröbe

et al. 2024

Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

The zero-shot effectiveness of neural retrieval models is often evaluated on the BEIR benchmark-a combination of different IR evaluation datasets. Interestingly, previous studies found that particularly on the BEIR subset Touché 2020, an argument retrieval task, neural retrieval models are considerably less effective than BM25. Still, so far, no further investigation has been conducted on what makes argument retrieval so "special". To more deeply analyze the respective potential limits of neural retrieval models, we run a reproducibility study on the Touché 2020 data. In our study, we focus on two experiments: (i) a black-box evaluation (i.e., no model retraining), incorporating a theoretical exploration using retrieval axioms, and (ii) a data denoising evaluation involving post-hoc relevance judgments. Our black-box evaluation reveals an inherent bias of neural models towards retrieving short passages from the Touché 2020 data, and we also find that quite a few of the neural models' results are unjudged in the Touché 2020 data. As many of the short Touché passages are not argumentative and thus non-relevant per se, and as the missing judgments complicate fair comparison, we denoise the Touché 2020 data by excluding very short passages (less than 20 words) and by augmenting the unjudged data with post-hoc judgments following the Touché guidelines. On the denoised data, the effectiveness of the neural models improves by up to 0.52 in nDCG@10, but BM25 is still more effective. Our code and the augmented Touché 2020 dataset are available at https://github.com/castorini/touche-error-analysis. CCS CONCEPTS• Information systems → Retrieval models and ranking; Evaluation of retrieval results.

show abstract

Section: Modelsupporting

confidence: 56%

Section: Improving Effectiveness At Inference Timementioning

confidence: 99%