Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models

Abstract

Multi-encoder models are a broad family of context-aware Neural Machine Translation (NMT) systems that aim to improve translation quality by encoding document-level contextual information alongside the current sentence. The context encoding is undertaken by contextual parameters, trained on documentlevel data. In this work, we show that training these parameters takes large amount of data, since the contextual training signal is sparse. We propose an efficient alternative, based on splitting sentence pairs, that allows to enrich the training signal of a set of parallel sentences by breaking intra-sentential syntactic links, and thus frequently pushing the model to search the context for disambiguating clues. We evaluate our approach with BLEU and contrastive test sets, showing that it allows multi-encoder models to achieve comparable performances to a setting where they are trained with ×10 document-level data. We also show that our approach is a viable option to context-aware NMT for language pairs with zero document-level parallel data.

Towards Formality-Aware Neural Machine Translation by Leveraging Context Information

“…Since a few tokens within a sentence reveal the formality, context-aware NMT models have difficulty to training and evaluating consistent stylistic translation. First, due to this sparsity (Lupo et al, 2021) that also occurs within the context, a training alternative is needed to amplify the weak training signal. In addition, BLEU (Papineni et al, 2002), a standard translation quality metric, often fails to capture improvements, which bring subtle differences.…”

“…Since recent context-aware NMT models (Maruf et al, 2019;Bao et al, 2021) incorporate the context into the NMT task, these models can be utilized to infer the appropriate formality from the context and, accordingly, generate consistent translations. However, due to the sparsity of context (Lupo et al, 2021), it is challenging to capture a few formality-related words scattered within the context. Yin et al (2021) guides the model to concentrate on the relevant contexts by collecting human-annotated contextual dataset, but this method resorts to manual annotations, which is time-consuming and labor-intensive.…”

Scaling Law for Document Neural Machine Translation

“…Miculicich et al (2018) pro-posed a hierarchical attention mechanism to capture the discourse information while Maruf et al (2019) employed a selective attention module to select the most relevant information in the context. Lupo et al (2021) further improved these methods by splitting the sentence into smaller segments to overcome the training sparsity problem. Recently, Tiedemann and Scherrer (2017); Ma et al (2020) suggest that Transformer has the ability to translate multiple sentences directly, and this documentby-document paradigm further reduce the context related errors.…”

Addressing the Length Bias Challenge in Document-Level Neural Machine Translation

“…Document-level neural machine translation can be broadly divide into two categories, including sentence-to-sentence (sen2sen) approach and document-to-document (doc2doc) approach (Maruf et al, 2021). The former feed the context as additional information to assist the translation of each single sentence in the document independently, which is also known as multi encoder method (Lupo et al, 2022) However, the scarcity of the datasets and the sparsity of the contextual information make these model hard to be trained. Lupo et al (2022) further address this problem by splitting the sentence into smaller pieces to augment the document-level corpus.…”