Exploiting Curriculum Learning in Unsupervised Neural Machine Translation

Abstract

Back-translation (BT) has become one of the de facto components in unsupervised neural machine translation (UNMT), and it explicitly makes UNMT have translation ability. However, all the pseudo bi-texts generated by BT are treated equally as clean data during optimization without considering the quality diversity, leading to slow convergence and limited translation performance. To address this problem, we propose a curriculum learning method to gradually utilize pseudo bi-texts based on their quality from multiple granularities. Specifically, we first apply crosslingual word embedding to calculate the potential translation difficulty (quality) for the monolingual sentences. Then, the sentences are fed into UNMT from easy to hard batch by batch. Furthermore, considering the quality of sentences/tokens in a particular batch are also diverse, we further adopt the model itself to calculate the fine-grained quality scores, which are served as learning factors to balance the contributions of different parts when computing loss and encourage the UNMT model to focus on pseudo data with higher quality. Experimental results on WMT 14 En↔Fr, WMT 16 En↔De, WMT 16 En↔Ro, and LDC En↔Zh translation tasks demonstrate that the proposed method achieves consistent improvements with faster convergence speed. 1

M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis

“…Curriculum Learning. The concept of Curriculum Learning (CL) is initially introduced by (Bengio et al, 2009), and has since manifested its multifaceted advantages across a diverse array of machine learning tasks (Platanios et al, 2019;Wang et al, 2019;Xu et al, 2020;Lu and Zhang, 2021). One of the advantages of CL is denoising, which is achieved by training more time on clean data and less time on noisy data .…”