Effective Parallel Corpus Mining using Bilingual Sentence Embeddings

Abstract

This paper presents an effective approach for parallel corpus mining using bilingual sentence embeddings. Our embedding models are trained to produce similar representations exclusively for bilingual sentence pairs that are translations of each other. This is achieved using a novel training method that introduces hard negatives consisting of sentences that are not translations but that have some degree of semantic similarity. The quality of the resulting embeddings are evaluated on parallel corpus reconstruction and by assessing machine translation systems trained on gold vs. mined sentence pairs. We find that the sentence embeddings can be used to reconstruct the United Nations Parallel Corpus (Ziemski et al., 2016) at the sentence level with a precision of 48.9% for en-fr and 54.9% for enes. When adapted to document level matching, we achieve a parallel document matching accuracy that is comparable to the significantly more computationally intensive approach of Uszkoreit et al. (2010). Using reconstructed parallel data, we are able to train NMT models that perform nearly as well as models trained on the original data (within 1-2 BLEU).

Hierarchical Document Encoder for Parallel Corpus Mining

“…On a test set mined from the web, all models achieve strong retrieval performance, the best being 91.4% P@1 for en-fr and 81.8% for en-es from the hierarchical document models. On the United Nations (UN) document mining task (Ziemski et al, 2016), our best model achieves 96.7% P@1 for en-fr and 97.3% P@1 for en-es, a 3%+ absolute improvement over the prior state-of-the-art (Guo et al, 2018;Uszkoreit et al, 2010). We also evaluate on a noisier version of the UN task where we do not have the ground truth sentence alignments from the original corpus.…”

“…Previous work on parallel document mining using large distributed systems has proven effective (Uszkoreit et al, 2010;Antonova and Misyurev, 2011), but these systems are often heavily engineered and computationally intensive. Recent work on parallel data mining has focused on sentence-level embeddings (Guo et al, 2018;Artetxe and Schwenk, 2018;Yang et al, 2019). However, these sentence embedding methods have had limited success when applied to documentlevel mining tasks (Guo et al, 2018).…”

“…Recent work on parallel data mining has focused on sentence-level embeddings (Guo et al, 2018;Artetxe and Schwenk, 2018;Yang et al, 2019). However, these sentence embedding methods have had limited success when applied to documentlevel mining tasks (Guo et al, 2018). A recent study from Yang et al (2019) shows that document embeddings obtained from averaging sentence embeddings can achieve state-of-the-art performance in document retrieval on the United Nation (UN) corpus.…”

Simple and Effective Paraphrastic Similarity from Parallel Translations

“…Baselines. We compare to several models from prior work (Guo et al, 2018;Chidambaram et al, 2018). A fair comparison to other models is difficult due to different training setups.…”

“…Most previous work for cross-lingual representations has focused on models based on encoders from neural machine translation (Espana-Bonet et al, 2017;Schwenk and Douze, 2017;Schwenk, 2018) or deep architectures using a contrastive loss (Grégoire and Langlais, 2018;Guo et al, 2018;Chidambaram et al, 2018). However, the paraphrastic sentence embedding literature has observed that simple models such as pooling word embeddings generalize significantly better than complex architectures (Wieting et al, 2016b).…”