Shallow pooling for sparse labels

Abstract

Recent years have seen enormous gains in core information retrieval tasks, including document and passage ranking. Datasets and leaderboards, and in particular the MS MARCO datasets, illustrate the dramatic improvements achieved by modern neural rankers. When compared with traditional information retrieval test collections, such as those developed by TREC, the MS MARCO datasets employ substantially more queries -thousands vs. dozens -with substantially fewer known relevant items per query -often just one. For example, 94% of the nearly seven thousand queries in the MS MARCO passage ranking development set have only a single known relevant passage, and no query has more than four. Given the sparsity of these relevance labels, the MS MARCO leaderboards track improvements with mean reciprocal rank (MRR). In essence, a relevant item is treated as the "right answer", with rankers scored on their ability to place this item as high in the ranking as possible. In working with these sparse labels, we have observed that the top items returned by a ranker often appear superior to judged relevant items. Others have reported the same observation.To test this observation, we employed crowdsourced workers to make preference judgments between the top item returned by a modern neural ranking stack and a judged relevant item for the nearly seven thousand queries in the passage ranking development set. The results support our observation. If we imagine a perfect ranker under MRR, with a score of 1 on all queries, our preference judgments indicate that a searcher would prefer the top result from a modern neural ranking stack more frequently than the top result from the imaginary perfect ranker, making our neural ranker "better than perfect".To understand the implications for the leaderboard, we pooled the top document from available runs near the top of the passage ranking leaderboard for over 500 queries. We employed crowdsourced workers to make preference judgments over these pools and re-evaluated the runs. Our results support our concerns that current MS MARCO datasets may no longer be able to recognize genuine improvements in rankers. In future, if rankers are measured against a single "right answer", this answer should be the best answer or most preferred answer, and maintained with ongoing judgments. Since only the best answer is required, this ongoing maintenance might be performed with shallow pooling. When a previously unjudged document is surfaced as the top item in a ranking, it can directly compared with the previous known best answer.

Hindsight: Posterior-guided training of retrievers for improved open-ended generation

“…Shifting our attention to MS MARCO NLGen, we notice that ELBoLoss still outperforms MarginalizedLoss by 1-2 points across our metrics, reflecting smaller-but nonetheless consistentgains when compared with the one-to-many generation task. We also observe that success@1 is lower for both methods, when compared with Wizard of Wikipedia, a fact we largely attributebased on manual inspection-to the presence of many false negatives, i.e., passages that contain the answer but aren't marked as gold passages, which is consistent with the findings from related studies Arabzadeh et al (2021). Overall, we find that ELBOLOSS improves relevance of retrieved passages over MARGINALIZEDLOSS for two qualitatively different tasks, with larger gains for the one-to-many generation task.…”

A Sensitivity Analysis of the MSMARCO Passage Collection

“…The work we report here was carried out in the period May-August 2021, and was conceived and executed independently of and concurrently with the complementary work of Arabzadeh et al [1].…”

DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine

“…Reducing False Negatives False negatives are a common issue existing in the passage retrieval datasets (Qu et al, 2021;Arabzadeh et al, 2021), which denotes there are passages relevant to the query but labelled as negative. In this section, we discuss our strategy for reducing the false negatives in the development and testing sets of DuReader retrieval .…”

“…• Arabzadeh et al (2021) and Qu et al (2021) observe that false negatives (i.e. relevant passages but labelled as negatives) are a common issue existing in the passage retrieval datasets due to their large scale but limited human annotation.…”