Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing

Abstract

We describe models focused at the understudied problem of translating between monolingual and code-mixed language pairs. More specifically, we offer a wide range of models that convert monolingual English text into Hinglish (code-mixed Hindi and English). Given the recent success of pretrained language models, we also test the utility of two recent Transformer-based encoder-decoder models (i.e., mT5 and mBART) on the task finding both to work well. Given the paucity of training data for code-mixing, we also propose a dependency-free method for generating code-mixed texts from bilingual distributed representations that we exploit for improving language model performance. In particular, armed with this additional data, we adopt a curriculum learning approach where we first finetune the language models on synthetic data then on gold code-mixed data. We find that, although simple, our synthetic code-mixing method is competitive with (and in some cases is even superior to) several standard methods (backtranslation, method based on equivalence constraint theory) under a diverse set of conditions. Our work shows that the mT5 model, finetuned following the curriculum learning procedure, achieves best translation performance (12.67 BLEU). Our models place first in the overall ranking of the English-Hinglish official shared task.

Enhancing Fake News Detection in Romanian Using Transformer-Based Back Translation Augmentation

“…The results demonstrate the effectiveness of BT augmentation in improving performance for text classification tasks. This finding is consistent with previous research [36,37], which demonstrated the efficacy of utilizing pre-trained models like mBART for BT augmentation. By leveraging the pre-trained language modeling capabilities of mBART, we can generate high-quality synthetic translations that can be used to augment the training data.…”

Can You Translate for Me? Code-Switched Machine Translation with Large Language Models

“…Due to resource scarcity, there have been efforts to explore the utilization of back-translated data in NMT (Jawahar et al, 2021). There are multiple efforts towards fine-tuning pre-trained language models, and generating pseudo parallel data (Winata et al, 2019;Gautam et al, 2021;Jawahar et al, 2021;Srivastava and Singh, 2022;Solorio et al, 2021). However, the use of large language models is still unexplored for CSMT.…”

AraT5: Text-to-Text Transformers for Arabic Language Generation

“…There is rising interest in translating code-switched data (Nagoudi et al, 2021). Our purpose here is to translate Arabic text involving code-switching from a foreign language into (i) that foreign language as well as into (ii) MSA.…”

“…Our work also meets an existing need for pre-trained Transformer-based sequenceto-sequence models. In other words, while several BERT-based models have been pre-trained for Arabic (Antoun et al, 2020;Abdul-Mageed et al, 2021;Inoue et al, 2021), no such attempts have been made to create sequence-to-sequence models that we know of. Another motivation for our work is absence of an evaluation benchmark for Arabic language generation tasks.…”

“…Although pre-trained with ∼ 49% less data, our new models perform significantly better than mT5 on all ARGEN tasks (in 52 out of 59 test sets) and set several new SOTAs. Our models also establish new SOTA on the recently-proposed, large Arabic language understanding evaluation benchmark ARLUE (Abdul-Mageed et al, 2021). Our models are publicly available.…”