Multilingual Translation via Grafting Pre-trained Language Models

Abstract

Can pre-trained BERT for one language and GPT for another be glued together to translate texts? Self-supervised training using only monolingual data has led to the success of pretrained (masked) language models in many NLP tasks. However, directly connecting BERT as an encoder and GPT as a decoder can be challenging in machine translation, for GPT-like models lack a cross-attention component that is needed in seq2seq decoders. In this paper, we propose Graformer to graft separately pre-trained (masked) language models for machine translation. With monolingual data for pre-training and parallel data for grafting training, we maximally take advantage of the usage of both types of data. Experiments on 60 directions show that our method achieves average improvements of 5.8 BLEU in x2en and 2.9 BLEU in en2x directions comparing with the multilingual Transformer of the same size 1 .

RedApt: An Adaptor for wav2vec 2 EncodingFaster and Smaller Speech Translation without Quality Compromise

“…Similar to the modern ST architectures (Gállego et al, 2021;, we use pretrained W2V2 large as our encoder and pretrained mBART50 decoder as the decoder. We randomly initialize the top 3 layers of W2V2 in experiments involving RedApt and find that it enables faster convergence, verifying earlier observations by Sun et al (2021). We free W2V2 feature extractor.…”

BERT-NAR-BERT: A Non-Autoregressive Pre-Trained Sequence-to-Sequence Model Leveraging BERT Checkpoints

“…Sun et al [19] proposed a grafting approach, where they stack multiple sets of BERT encoder layers that follow autoregressive manner and GPT decoder decoder layers, and freeze selections of encoder or decoder parameters during fine-tuning. This approach leads to strong gains in output quality, however, they do not compare model sizes, training or inference time directly with previous work.…”

M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation

“…Pre-training a text decoder can be done independently (e.g., GPT2 [26]) or jointly with an encoder for sequence-to-sequence tasks (e.g., mT5 [27], mBart [6]). With the former approach, after the pre-training phase, the text decoder needs to be fused with additional encoders or adapters [28], which increases the complexity of architectures. For the latter approach, the decoder component can usually be used as an individual module without architectural modifications.…”