Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

Zhang, Xinyu; Ma, Xueguang; Shi, Peng; Lin, Jimmy

doi:10.48550/arxiv.2108.08787

Cited by 7 publications

(12 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Hence, we adopt XOR-QA [Asai et al, 2020] dataset and Mr. TYDI [Zhang et al, 2021] dataset to evaluate our method on the two settings. Both of the two datasets are constructed from TYDI, a question answering dataset covering eleven typologically diverse languages.…”

Section: Cross Lingual Query Passage Retrievalmentioning

confidence: 99%

“…The Mr. TYDI dataset is a multi-lingual benchmark dataset for mono-lingual query passage retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. Same to the source paper [Zhang et al, 2021], we use MRR@100 and Recall@100 as metrics.…”

Section: Cross Lingual Query Passage Retrievalmentioning

confidence: 99%

“…In this paper, we adopt a zero-shot setting to evaluate our method. We train the pre-trained model on Natural Question data and directly test the model on XOR-QA [Asai et al, 2020] dataset and Mr. TYDI [Zhang et al, 2021] dataset. We train the model on 8 NVIDIA Tesla V100 GPUs (with 32GB RAM).…”

Section: Cross Lingual Query Passage Retrievalmentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval

Wu¹,

Liang²,

Houxing³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent research demonstrates the effectiveness of using pretrained language models (PLM) to improve dense retrieval and multilingual dense retrieval. In this work, we present a simple but effective monolingual pretraining task called contrastive context prediction (CCP) to learn sentence representation by modeling sentence level contextual relation. By pushing the embedding of sentences in a local context closer and pushing random negative samples away, different languages could form isomorphic structure, then sentence pairs in two different languages will be automatically aligned. Our experiments show that model collapse and information leakage are very easy to happen during contrastive training of language model, but language-specific memory bank and asymmetric batch normalization operation play an essential role in preventing collapsing and information leakage, respectively. Besides, a post-processing for sentence embedding is also very effective to achieve better retrieval performance. On the multilingual sentence retrieval task Tatoeba, our model achieves new SOTA results among methods without using bilingual data. Our model also shows larger gain on Tatoeba when transferring between non-English pairs. On two multi-lingual querypassage retrieval tasks, XOR Retrieve and Mr.TYDI, our model even achieves two SOTA results in both zero-shot and supervised setting among all pretraining models using bilingual data. The pretrained model and code are available in this link:https: //github.com/wuning0929/CCP_IJCAI22.

show abstract

Section: Cross Lingual Query Passage Retrievalmentioning

confidence: 99%

Section: Cross Lingual Query Passage Retrievalmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval

Wu¹,

Liang²,

Houxing³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…First, in the passage retrieval step, we replace mBERT with mLUKE (Ri et al, 2021). Second, we construct sparse indices from which we will retrieve passages to augment dense retriever-retrieved passages, inspired by Zhang et al (2021) but uses a different densesparse hybrid approach. Finally, we encode each question and passage independently as opposed to all passages together following the Fusion-in-Decoder (Izacard and Grave, 2020) approach.…”

Section: System Architecture and Pipelinementioning

confidence: 99%

MIA 2022 Shared Task Submission: Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering

Tu¹,

Padmanabhan²

2022

Proceedings of the Workshop on Multilingual Information Access (MIA)

View full text Add to dashboard Cite

We describe our two-stage system for the Multilingual Information Access (MIA) 2022 Shared Task on Cross-Lingual Open-Retrieval Question Answering. The first stage consists of multilingual passage retrieval with a hybrid dense and sparse retrieval strategy. The second stage consists of a reader which outputs the answer from the top passages returned by the first stage. We show the efficacy of using entity representations, sparse retrieval signals to help dense retrieval, and Fusion-in-Decoder. On the development set, we obtain 43.46 F1 on XOR-TyDi QA and 21.99 F1 on MKQA, for an average F1 score of 32.73. On the test set, we obtain 40.93 F1 on XOR-TyDi QA and 22.29 F1 on MKQA, for an average F1 score of 31.61. We improve over the official baseline by over 4 F1 points on both the development and test sets. 1

show abstract

“…-Judgment pools were retrieved using older systems. New neural systems are thus more likely to systematically identify relevant unjudged documents [38,40,46]. -Many of the early test collections have only binary judgments.…”

Section: Introductionmentioning

confidence: 99%

HC4: A New Suite of Test Collections for Ad Hoc CLIR

Lawrie¹,

Mayfield²,

Oard³

et al. 2022

Preprint

View full text Add to dashboard Cite

HC4 is a new suite of test collections for ad hoc Cross-Language Information Retrieval (CLIR), with Common Crawl News documents in Chinese, Persian, and Russian, topics in English and in the document languages, and graded relevance judgments. New test collections are needed because existing CLIR test collections built using pooling of traditional CLIR runs have systematic gaps in their relevance judgments when used to evaluate neural CLIR methods. The HC4 collections contain 60 topics and about half a million documents for each of Chinese and Persian, and 54 topics and five million documents for Russian. Active learning was used to determine which documents to annotate after being seeded using interactive search and judgment. Documents were judged on a three-grade relevance scale. This paper describes the design and construction of the new test collections and provides baseline results for demonstrating their utility for evaluating systems.

show abstract

Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

Cited by 7 publications

References 21 publications

Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval

Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval

MIA 2022 Shared Task Submission: Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering

HC4: A New Suite of Test Collections for Ad Hoc CLIR

Contact Info

Product

Resources

About