Dual-level interactive multimodal-mixup encoder for multi-modal neural machine translation

Ye, Junjie; Guo, Junjun

doi:10.1007/s10489-022-03331-8

Cited by 8 publications

(1 citation statement)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous research on Russian translation focuses on direct translation and machine translation, analyzes the Russian language from a linguistic point of view, focuses on the analysis of the text itself, and less on the meanings constructed by other modalities such as image modality, sound modality, and so on, in translation, and ignores the influence of modal interaction on translation [1][2][3]. However, with the development of multimodal machine translation, if the Russian text is placed in the translation process of each modality independently or if its translation is combined with the theory, then the Russian translation will not be able to achieve the desired results [4][5][6][7]. Therefore, the previous research model is no longer adapted to today's multimodal Russian translation research, and it is necessary to break through the previous research perspectives and frameworks [8][9][10].…”

Section: Introductionmentioning

confidence: 99%

Effective application of multimodal discourse analysis in Russian translation

Wu,

Zhang,

Zhang

2024

Applied Mathematics and Nonlinear Sciences

View full text Add to dashboard Cite

Based on ELAN multimodal discourse analysis software, this paper constructs a multimodal Russian translation model based on the machine translation model with visual grammar and multimodal discourse analysis as the theoretical basis. To address the issue of missing semantics caused by insufficient input information at the source of real-time translation, the model uses images as auxiliary modalities. The real-time Russian translation model is constructed using the wait-k strategy and the concept of multimodal self-attention. Experiments and analysis are carried out on the Multi30k training set, and the generalization ability and translation effect of the model are finally evaluated with the test set. The results show that by applying multimodal discourse analysis to Russian translation, the three translation evaluation indexes of BLEU, METEOR, and TER are improved by 1.3, 1.0, and 1.4 percentage points, respectively, and the phenomenon of phantom translation is effectively reduced.

show abstract