Machine Translation Evaluation and Optimization

Dorr, Bonnie J.; Olive, Joseph P.; McCary, John; Christianson, Caitlin

doi:10.1007/978-1-4419-7713-7_5

Cited by 13 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation of MT systems is an essential area of research, both for assessing the effectiveness of existing MT systems and improving MT systems' performance (Dorr et al, 2011). Several evaluation methods of MT have been used over the phase of Machine Translation Evaluation (MTE).…”

Section: Mt Evaluation Methodsmentioning

confidence: 99%

Evaluation of Google Image Translate in Rendering Arabic Signage into English

Almahasees

Mahmoud²

2022

WJEL

View full text Add to dashboard Cite

When people travel to another country for work or leisure, they regularly need a medium to help them understand the written messages in other languages. Google Translate offers a new service: translating the content of images (texts) instantly and freely into 100 languages powered by the Neural Machine Translation approach (NMT). In this vein, the current research paper attempts to evaluate the accuracy of Google Image Translate service in rendering the texts printed on Arabic signage: banners and road and shop signs from Arabic into English. Besides, it aims to identify the capacity of Google Translate in rendering Arabic signage into English effectively without the help of human translators. The paper adopts the Linguistic Error Analysis Framework of Costa et al. (2015) in analyzing the output of Google image service in terms of orthography, grammar, lexis, and semantics. The paper shows that Google Translate made the following errors while rendering the content of images into English: mistranslation, omission, additions, wrong choice, misordering, subject-verb disagreement, and semantic errors. In conclusion, the Google Image Translate service helps the users configure the gist of the image. However, a human translator is still needed since MT may not provide an adequate and effective translation as humans do.

show abstract

Section: Mt Evaluation Methodsmentioning

confidence: 99%

Evaluation of Google Image Translate in Rendering Arabic Signage into English

Almahasees

Mahmoud²

2022

WJEL

View full text Add to dashboard Cite

show abstract

“…e biggest issue that manual intrinsic methods face is their subjectivity and nonreproducibility, apart from their price and timeliness. Automatic intrinsic measures, such as ranking, compute sentence similarity among MT outputs and a fixed set of references to produce rankings among MT systems [35]. Unlike intrinsic measures which are focused on accuracy and text coherence, extrinsic methods focus on the effectiveness or usability of MT output in terms of the specific task such as PE [36][37][38].…”

Section: Evaluation Of Mt Systemmentioning

confidence: 99%

MT Evaluation in the Context of Language Complexity

et al. 2021

View full text Add to dashboard Cite

The paper focuses on investigating the impact of artificial agent (machine translator) on human agent (posteditor) using a proposed methodology, which is based on language complexity measures, POS tags, frequent tagsets, association rules, and their summarization. We examine this impact from the point of view of language complexity in terms of word and sentence structure. By the proposed methodology, we analyzed 24 733 tags of English to Slovak translations of technical texts, corresponding to the output of two MT systems (Google Translate and the European Commission’s MT tool). We used both manual (adequacy and fluency) and semiautomatic (HTER metric) MT evaluation measures as the criteria for validity. We show that the proposed methodology is valid based on the evaluation of frequent tagsets and rules of MT outputs produced by Google Translate or of the European Commission’s MT tool, and both postedited MT (PEMT) outputs using baseline methods. Our results have also shown that PEMT output produced by Google Translate is characterized by more frequent tagsets such as verbs in the infinitive with modal verbs compared to its MT output, which is characterized by masculine, inanimate nouns in locative of singular. In the MT output, produced by the European Commission’s MT tool, the most frequent tagset was verbs in the infinitive compared to its postedited MT output, where verbs in imperative and the second person of plural occurred. These findings are also obtained from the use of the proposed methodology for MT evaluation. The contribution of the proposed methodology is an identification of systematic not random errors. Additionally, the study can also serve as information for optimizing the translation process using postediting.

show abstract

“…As another example, Conroy and Dang (2008) have highlighted the downsides of ignoring linguistic quality while focusing on summary content during system evaluation. Additionally, the need for linguistic quality evaluation has been underscored in Dorr et al (2011); Graham et al (2013); Novikova et al (2017); Way (2018); Specia and Shah (2018).…”

Section: Evaluating Linguistic Qualitymentioning

confidence: 99%

GRUEN for Evaluating Linguistic Quality of Generated Text

Zhu¹,

Bhat²

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Automatic evaluation metrics are indispensable for evaluating generated text. To date, these metrics have focused almost exclusively on the content selection aspect of the system output, ignoring the linguistic quality aspect altogether. We bridge this gap by proposing GRUEN for evaluating Grammaticality, non-Redundancy, focUs, structure and coherENce of generated text. 1 GRUEN utilizes a BERTbased model and a class of syntactic, semantic, and contextual features to examine the system output. Unlike most existing evaluation metrics which require human references as an input, GRUEN is reference-less and requires only the system output. Besides, it has the advantage of being unsupervised, deterministic, and adaptable to various tasks. Experiments on seven datasets over four language generation tasks show that the proposed metric correlates highly with human judgments. 2

show abstract

Machine Translation Evaluation and Optimization

Cited by 13 publications

References 18 publications

Evaluation of Google Image Translate in Rendering Arabic Signage into English

Evaluation of Google Image Translate in Rendering Arabic Signage into English

MT Evaluation in the Context of Language Complexity

GRUEN for Evaluating Linguistic Quality of Generated Text

Contact Info

Product

Resources

About