Comparative Quality Estimation for Machine Translation Observations on Machine Learning and Features

Abstract

A deeper analysis on Comparative Quality Estimation is presented by extending the stateof-the-art methods with adequacy and grammatical features from other Quality Estimation tasks. The previously used linear method, unable to cope with the augmented features, is replaced with a boosting classifier assisted by feature selection. The methods indicated show improved performance for 6 language pairs, when applied on the output from MT systems developed over 7 years. The improved models compete better with reference-aware metrics.Notable conclusions are reached through the examination of the contribution of the features in the models, whereas it is possible to identify common MT errors that are captured by the features. Many grammatical/fluency features have a good contribution, few adequacy features have some contribution, whereas source complexity features are of no use. The importance of many fluency and adequacy features is language-specific.

Sentence-level quality estimation by predicting HTER as a multi-component metric

“…The German-English model got optimal with 300 hidden units. The English-German was trained using the additional 52 features from Avramidis (2017) which gave good development results only with 3,000 hidden units, which is an indication of overfitting. Table 8: Scores for the submitted models and for their corrected versions after the submission English-German A comparison of the models developed before the submission and the corrected ones are shown in Tables 7 and 8.…”

Method of measuring the effort related to post-editing machine translated outputs produced in the English>Polish language pair by Google, Microsoft and DeepL MT engines: A pilot study

“…Such a task requires the establishment of a reliable, reproducible and cost-efficient study framework, which could be used to measure the efficiency of various solutions. The framework proposed in this article is based on several studies conducted in the past (Avramidis 2017, Graham et al 2017, Han et al 2017, Fiederer and O'Brien 2009, Callison-Burch et al 2007, Snover et al 2006 and is adjusted to the project's specific needs. The pilot study described below was designed in such a way to determine whether the proposed method can be efficiently used to obtain reliable results related to post-editing time and effort.…”