Machine Translation of Restaurant Reviews: New Corpus for Domain Adaptation and Robustness

Abstract

We share a French-English parallel corpus of Foursquare restaurant reviews, and define a new task to encourage research on Neural Machine Translation robustness and domain adaptation, in a real-world scenario where better-quality MT would be greatly beneficial. We discuss the challenges of such usergenerated content, and train good baseline models that build upon the latest techniques for MT robustness. We also perform an extensive evaluation (automatic and human) that shows significant improvements over existing online systems. Finally, we propose taskspecific metrics based on sentiment analysis or translation accuracy of domain-specific polysemous words.

PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents

“…The advancement of Neural Machine Translation (NMT) has brought great improvement in translation quality when translating clean input, such as text from the news domain (Luong et al, 2015;Vaswani et al, 2017), and it was recently claimed that NMT has even achieved human parity in certain language pairs (Hassan et al, 2018;Barrault et al, 2019). Despite its remarkable advancements, the applicability of NMT over User-Generated Contents (UGC), such as social media text, still remains limited (Michel and Neubig, 2018;Berard et al, 2019a). Since UGC are prevailing in our real-life communication, it is undoubtedly one of the challenges we need to overcome to make MT systems invaluable for promoting cross-cultural communication.…”

Evaluating Robustness to Input Perturbations for Neural Machine Translation

“…Previous work has used translation quality measures such as BLEU on noisy input as an indicator of robustness. Absolute model performance on noisy input is important, and we believe this is an appropriate measure for noisy domain evaluation (Michel and Neubig, 2018;Berard et al, 2019;. However, it does not disentangle model quality from the relative degradation under added noise.…”

“…EN↔DE and EN↔FI models are trained with pre-processed WMT18 news data and tested with the latest news test sets (new-stest2019). Recently, two datasets were built from usergenerated content, MTNT (Michel and Neubig, 2018) and 4SQ (Berard et al, 2019 baseline models are trained with aggregated data of Europarl-v7 (Koehn, 2005), NewsCommentary-v14 (Bojar et al, 2018), OpenSubtitles-v2018 (Lison and Tiedemann, 2016), and ParaCrawl-v5 1 , which simulates the UGC training corpus used in 4SQ benchmarks, and they are tested with the latest WMT new test sets supporting EN↔FR (new-stest2014).…”

“…The first is synthetic misspelling: each word is misspelled with probability of 0.1, and the strategy is randomly chosen from single-character deletion, insertion, and substitution (Karpukhin et al, 2019). The second perturbation is letter case changing: each sentence is modified with probability of 0.5, and the strategy is randomly chosen from upper-casing all letters, lower-casing all letters, and title-casing all words (Berard et al, 2019). 2 Since we change the letter case in the test data, we always report case-insensitive BLEU with '13a' tokenization using sacreBLEU (Post, 2018).…”

Synthesizing Parallel Data of User-Generated Texts with Zero-Shot Neural Machine Translation

“…Berard et al (2019a) showed that a large monolingual corpus of UGT can be successfully back-translated with a system trained on P R1-R2 parallel data.…”