How Deep is your Learning: the DL-HARD Annotated Deep Learning Dataset

Mackie, Iain D.; Dalton, Jeff; Yates, Andrew

doi:10.1145/3404835.3463262

Cited by 26 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This suggests that location queries are over-represented in MS MARCO, and that, in general, the queries that are discarded come from a different distribution than those that survive. This distribution shift is potentially concerning, given that others have found description queries to be among the more challenging and interesting of the query types [18].…”

Section: Analysis Of Discarded Queriesmentioning

confidence: 99%

On Survivorship Bias in MS MARCO

Gupta

MacAvaney

2022

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

Section: Analysis Of Discarded Queriesmentioning

confidence: 99%

On Survivorship Bias in MS MARCO

Gupta

MacAvaney

2022

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…those that "require interpretation [...] beyond the immediate meaning of the terms" [1] in the queries). Coincidentally, Mackie et al [25] introduced a framework for identifying hard queries that challenge DRs. Sciavolino et al [34] found that DRs also perform poorly, and worse than traditional bag-of-words methods, on queries that contain entities.…”

Section: Related Workmentioning

confidence: 99%

CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos

Zhuang,

Zuccon

2022

Preprint

View full text Add to dashboard Cite

Current dense retrievers are not robust to out-of-domain and outlier queries, i.e. their effectiveness on these queries is much poorer than what one would expect. In this paper, we consider a specific instance of such queries: queries that contain typos. We show that a small character level perturbation in queries (as caused by typos) highly impacts the effectiveness of dense retrievers. We then demonstrate that the root cause of this resides in the input tokenization strategy employed by BERT. In BERT, tokenization is performed using the BERT's WordPiece tokenizer and we show that a token with a typo will significantly change the token distributions obtained after tokenization. This distribution change translates to changes in the input embeddings passed to the BERT-based query encoder of dense retrievers. We then turn our attention to devising dense retriever methods that are robust to such queries with typos, while still being as performant as previous methods on queries without typos. For this, we use CharacterBERT as the backbone encoder and an efficient yet effective training method, called Self-Teaching (ST), that distills knowledge from queries without typos into the queries with typos. Experimental results show that CharacterBERT in combination with ST achieves significantly higher effectiveness on queries with typos compared to previous methods. Along with these results and the open-sourced implementation of the methods, we also provide a new passage retrieval dataset consisting of realworld queries with typos and associated relevance assessments on the MS MARCO corpus, thus supporting the research community in the investigation of effective and robust dense retrievers. Code, experimental results and dataset are made available at https://github. com/ielab/CharacterBERT-DR. CCS CONCEPTS• Information systems → Retrieval models and ranking.

show abstract

“…However, MS MARCO's annotation technique means that the queries tend to be artificially easy and exhibit undesirable qualities like the 'maximum passage bias' [20]. TREC Deep Learning [8] extends MS MARCO with dense judgments to provide a more useful benchmark, and DL-HARD [26] develops a more challenging subset with annotations and metadata. CODEC differs from these datasets in terms of length of queries, i.e.…”

Section: Document Rankingmentioning

confidence: 99%