Comparative Analysis of Current Approaches to Quality Estimation for Neural Machine Translation

Eo, Sugyeong; Park, Chanjun; Moon, Hyeonseok; Seo, Jaehyung; Lim, Heuiseok

doi:10.3390/app11146584

“…The pre-training task does not have to be tied to QE. Large pre-trained language models have also been used for QE (Hu et al, 2020;Moura et al, 2020;Nakamachi et al, 2020;Eo et al, 2021). In contrast, our pre-training aims not only to acquire better sentence representations in general but specifically to acquire better sentence representations for translation quality score estimation.…”

Section: Related Workmentioning

confidence: 99%

Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Zouhar,

Dhuliawala,

Zhou

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference. Stateof-the-art QE systems based on pretrained language models have been achieving remarkable correlations with human judgements yet they are computationally heavy and require human annotations, which are slow and expensive to create. To address these limitations, we define the problem of metric estimation (ME) where one predicts the automated metric scores also without the reference. We show that even without access to the reference, our model can estimate automated metrics (ρ=60% for BLEU, ρ=51% for other metrics) at the sentence-level. Because automated metrics correlate with human judgements, we can leverage the ME task for pre-training a QE model. For the QE task, we find that pre-training on TER is better (ρ=23%) than training for scratch (ρ=20%).

show abstract

“…The pre-training task does not have to be tied to QE. Large pre-trained language models have also been used for QE (Hu et al, 2020;Moura et al, 2020;Nakamachi et al, 2020;Eo et al, 2021). In contrast, our pre-training aims not only to acquire better sentence representations in general but specifically to acquire better sentence representations for translation quality score estimation.…”

Section: Related Workmentioning

confidence: 99%

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Zouhar¹,

Dhuliawala²,

Zhou³

et al. 2023

Preprint

0

View full text Add to dashboard Cite

Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference. Stateof-the-art QE systems based on pretrained language models have been achieving remarkable correlations with human judgements yet they are computationally heavy and require human annotations, which are slow and expensive to create. To address these limitations, we define the problem of metric estimation (ME) where one predicts the automated metric scores also without the reference. We show that even without access to the reference, our model can estimate automated metrics (ρ=60% for BLEU, ρ=51% for other metrics) at the sentence-level. Because automated metrics correlate with human judgements, we can leverage the ME task for pre-training a QE model. For the QE task, we find that pre-training on TER is better (ρ=23%) than training for scratch (ρ=20%).

show abstract

“…After the deep neural network (also called deep learning) [17] became popular in various areas, such as machine translation [2,41], chatbot [42], unmanned aerial vehicle [43], person re-identification [44,45], multiple object tracking [46], image recognition [47,48] and signal processing [49], explainable AI received attention again. This time, people saw the power of deep learning and never doubted its ability.…”

Section: Deep Learningmentioning

confidence: 99%

“…Artificial intelligence (AI) is more intelligent than humans in a few fields, such as image recognition competitions, chess, Go, and intellectual television questions and answers [1][2][3][4][5][6]. Furthermore, lifelong machine learning (LML) [7][8][9][10][11][12] aims to make AI more effective [13][14][15].…”

Section: Introductionmentioning

confidence: 99%

Dual-Track Lifelong Machine Learning-Based Fine-Grained Product Quality Analysis

Hong

¹

,

Guan

²

,

Xue

³

et al. 2023

Applied Sciences

2

0

View full text Add to dashboard Cite

Artificial intelligence (AI) systems are becoming wiser, even surpassing human performances in some fields, such as image classification, chess, and Go. However, most high-performance AI systems, such as deep learning models, are black boxes (i.e., only system inputs and outputs are visible, but the internal mechanisms are unknown) and, thus, are notably challenging to understand. Thereby a system with better explainability is needed to help humans understand AI. This paper proposes a dual-track AI approach that uses reinforcement learning to supplement fine-grained deep learning-based sentiment classification. Through lifelong machine learning, the dual-track approach can gradually become wiser and realize high performance (while keeping outstanding explainability). The extensive experimental results show that the proposed dual-track approach can provide reasonable fine-grained sentiment analyses to product reviews and remarkably achieve a 133% promotion of the Macro-F1 score on the Twitter sentiment classification task and a 27.12% promotion of the Macro-F1 score on an Amazon iPhone 11 sentiment classification task, respectively.

show abstract

Comparative Analysis of Current Approaches to Quality Estimation for Neural Machine Translation

Cited by 8 publications

References 22 publications

Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Dual-Track Lifelong Machine Learning-Based Fine-Grained Product Quality Analysis

Contact Info

Product

Resources

About