Document-Level Machine Translation with Large Language Models

Abstract

Large language models (LLMs) such as Chat-GPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking documentlevel machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs' ability on discourse modeling. The study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of Chat-GPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evaluating on a number of benchmarks, we surprisingly find that LLMs have demonstrated superior performance and show potential to become a new paradigm for document-level translation: 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of human evaluation; 1 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evaluation of LLMs. 2 * Equal contribution. 1 The protocol employed in this work was approved by the Tencent Institutional Review Board (IRB).2 We release our data and annotations at https://github. com/longyuewangdcu/Document-MT-LLM.

Generative AI and Educators: Partnering in Using Open Digital Content for Transforming Education

“…The potential of GAI to successfully translate texts in different languages is confirmed by authors such as Wang et al (2023). However, from the responses in this study, it seems as if GAI tools, in this case ChatGPT, might currently be unable to translate texts in all languages.…”

“…Several authors (e.g. Jiao et al, 2023;Wang et al, 2023;Onal & Kulavuz-Onal, 2023) confirm the high proficiency of language models in translation tasks, which is mainly possible due to machine translation technologies (Deng & Yu, 2022). This is also an example of how technologies develop and improve their potential to evolve further because of technological advancements.…”