DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine

Abstract

In this paper, we present DuReader retrieval , a large-scale Chinese dataset for passage retrieval. DuReader retrieval contains more than 90K queries and over 8M unique passages from Baidu search. To ensure the quality of our benchmark and address the shortcomings in other existing datasets, we (1) reduce the false negatives in development and testing sets by pooling the results from multiple retrievers with human annotations, (2) and remove the semantically similar questions between training with development and testing sets. We further introduce two extra out-of-domain testing sets for benchmarking the domain generalization capability. Our experiment results demonstrate that DuReader retrieval is challenging and there is still plenty of room for the community to improve, e.g. the generalization across domains, salient phrase and syntax mismatch between query and paragraph and robustness. DuReader retrieval will be publicly available at https: //github.com/baidu/DuReader/ tree/master/DuReader-Retrieval