Iterative Training of Unsupervised Neural and Statistical Machine Translation Systems

Abstract

Recent work achieved remarkable results in training neural machine translation (NMT) systems in a fully unsupervised way, with new and dedicated architectures that only rely on monolingual corpora. However, previous work also showed that unsupervised statistical machine translation (USMT) performs better than unsupervised NMT (UNMT), especially for distant language pairs. To take advantage of the superiority of USMT over UNMT, and considering that SMT suffers from well-known limitations overcome by NMT, we propose to define UNMT as NMT trained with the supervision of synthetic parallel data generated by USMT. This way we can exploit USMT up to its limits while ultimately relying on full-fledged NMT models to generate translations. We show significant improvements in translation quality over previous work and also that further improvements can be obtained by alternatively and iteratively training USMT and UNMT. Without the need of a dedicated architecture for UNMT, our simple approach can straightforwardly benefit from any recent and future advances in supervised NMT. Our systems achieve a new state-of-the-art for unsupervised machine translation in all of our six translation tasks for five diverse language pairs, surpassing even supervised SMT or NMT in some tasks. Furthermore, our analysis shows how crucial the comparability between the monolingual corpora used for unsupervised training is in improving translation quality.

GRI: Graph-based Relative Isomorphism of Word Embedding Spaces

“…Other dominant factors identified in the literature that limit the end performance of BLI systems include: (i) linguistic differences (ii) algorithmic mismatch, (iii) variation in data size, (iv) parameterization etc. Similar to the supervised models, the unsupervised variants of BLI are also unable to cater to the above-mentioned challenges (Kim et al, 2020;Marie and Fujita, 2020). Instead of relying on embedding spaces trained completely independent of each other, in the recent past there have been a shift in explicitly using the isomorphism measures alongside distributional training objective (Marchisio et al, 2022).…”

GARI: Graph Attention for Relative Isomorphism of Arabic Word Embeddings

“…Some other noteworthy aspects identified in the literature that limit the end performance of the BLI systems, include: (a) algorithmic mismatch for independently trained monolingual embeddings, (b) different parameterization, (c) variable data sizes, (d) linguistic difference, etc., (Marie and Fujita, 2020;Marchisio et al, 2022).…”