Zipporah: a Fast and Scalable Data Cleaning System for Noisy
            Web-Crawled Parallel Corpora

Xu, Hainan; Koehn, Philipp

doi:10.18653/v1/d17-1319

Cited by 61 publications

(64 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Zipporah (Xu and Koehn, 2017;, which is often used as a baseline comparison, uses language model and word translation scores, with weights optimized to separate clean and synthetic noise data. In our setup, we trained Zipporah models for both language pairs Sinhala-English and Nepali-English.…”

Section: Other Similarity Methodsmentioning

confidence: 99%

Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings

Chaudhary¹,

Tang²,

Guzmán³

et al. 2019

Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)

Self Cite

View full text Add to dashboard Cite

In this paper, we describe our submission to the WMT19 low-resource parallel corpus filtering shared task. Our main approach is based on the LASER toolkit (Language-Agnostic SEntence Representations), which uses an encoder-decoder architecture trained on a parallel corpus to obtain multilingual sentence representations. We then use the representations directly to score and filter the noisy parallel sentences without additionally training a scoring function. We contrast our approach to other promising methods and show that LASER yields strong results. Finally, we produce an ensemble of different scoring methods and obtain additional gains. Our submission achieved the best overall performance for both the Nepali-English and Sinhala-English 1M tasks by a margin of 1.3 and 1.4 BLEU respectively, as compared to the second best systems. Moreover, our experiments show that this technique is promising for low and even no-resource scenarios.

show abstract

Section: Other Similarity Methodsmentioning

confidence: 99%

Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings

Chaudhary¹,

Tang²,

Guzmán³

et al. 2019

Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Data For additional unlabeled-domain data, we use web-crawled bitext from the Paracrawl project. 4 We filter the data using the Zipporah cleaning tool (Xu and Koehn, 2017), with a threshold score of 1. After filtering, we have around 13.6 million Paracrawl sentences available for German-English and 3.7 million Paracrawl sentences available for Russian-English.…”

Section: Unlabeled-domainmentioning

confidence: 99%

Curriculum Learning for Domain Adaptation in Neural Machine Translation

Zhang

Shapiro

Kumar

et al. 2019

Proceedings of the 2019 Conference of the North

120

View full text Add to dashboard Cite

We introduce a curriculum learning approach to adapt generic neural machine translation models to a specific domain. Samples are grouped by their similarities to the domain of interest and each group is fed to the training algorithm with a particular schedule. This approach is simple to implement on top of any neural framework or architecture, and consistently outperforms both unadapted and adapted baselines in experiments with two distinct domains and two language pairs.

show abstract

“…• Source and target language model scores • Hunalign scores (Varga et al, 2007) • Zipporah adequacy scores (Xu and Koehn, 2017), using a probabilistic bilingual dictionary computed on Europarl.…”

Section: Data Selectionmentioning

confidence: 99%

Lingua Custodia at WMT’19: Attempts to Control Terminology

Burlot¹

2019

Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)

View full text Add to dashboard Cite

This paper describes Lingua Custodia's submission to the WMT'19 news shared task for German-to-French on the topic of the EU elections. We report experiments on the adaptation of the terminology of a machine translation system to a specific topic, aimed at providing more accurate translations of specific entities like political parties and person names, given that the shared task provided no in-domain training parallel data dealing with the restricted topic. Our primary submission to the shared task uses backtranslation generated with a type of decoding allowing the insertion of constraints in the output in order to guarantee the correct translation of specific terms that are not necessarily observed in the data.

show abstract

Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora

Cited by 61 publications

References 2 publications

Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings

Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings

Curriculum Learning for Domain Adaptation in Neural Machine Translation

Lingua Custodia at WMT’19: Attempts to Control Terminology

Contact Info

Product

Resources

About