Argument Retrieval from Web

Shahshahani, Mahsa S.; Kamps, Jaap

doi:10.1007/978-3-030-58219-7_7

Cited by 2 publications

(1 citation statement)

References 6 publications

(5 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[58,65]. Previous works on argument retrieval [47,56] majorly made use of lexical retrieval models such as BM25 [53], DirichletLM [74], DPH [2], and TF-IDF [30]. These models were also commonly used to retrieve argumentative documents in argument search engines.…”

Section: Background and Related Workmentioning

confidence: 99%

Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR

Thakur,

Bonifacio,

Fröbe

et al. 2024

Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

The zero-shot effectiveness of neural retrieval models is often evaluated on the BEIR benchmark-a combination of different IR evaluation datasets. Interestingly, previous studies found that particularly on the BEIR subset Touché 2020, an argument retrieval task, neural retrieval models are considerably less effective than BM25. Still, so far, no further investigation has been conducted on what makes argument retrieval so "special". To more deeply analyze the respective potential limits of neural retrieval models, we run a reproducibility study on the Touché 2020 data. In our study, we focus on two experiments: (i) a black-box evaluation (i.e., no model retraining), incorporating a theoretical exploration using retrieval axioms, and (ii) a data denoising evaluation involving post-hoc relevance judgments. Our black-box evaluation reveals an inherent bias of neural models towards retrieving short passages from the Touché 2020 data, and we also find that quite a few of the neural models' results are unjudged in the Touché 2020 data. As many of the short Touché passages are not argumentative and thus non-relevant per se, and as the missing judgments complicate fair comparison, we denoise the Touché 2020 data by excluding very short passages (less than 20 words) and by augmenting the unjudged data with post-hoc judgments following the Touché guidelines. On the denoised data, the effectiveness of the neural models improves by up to 0.52 in nDCG@10, but BM25 is still more effective. Our code and the augmented Touché 2020 dataset are available at https://github.com/castorini/touche-error-analysis. CCS CONCEPTS• Information systems → Retrieval models and ranking; Evaluation of retrieval results.

show abstract