Investigation of Intelligibility Judgments

Reeder, Florence

doi:10.1007/978-3-540-30194-3_25

Cited by 2 publications

(3 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One previous investigation ( Östling and Tiedemann, 2017) used BLEU (Papineni et al, 2002) to determine that 70k sentences was sufficient to provide decent quality for a neural LRMTS. The assumption from LRMT developers is that including humans is expensive and time-consuming avoiding inclusion of more human-like measurements such as adequacy (Doherty, 2018), HTER (Snover et al, 2006), and fluency (Reeder, 2004). This article describes uses of those metrics along with the following others to help better overcome the evaluation challenge.…”

Section: Related Workmentioning

confidence: 99%

“…Range BLEU (Papineni et al, 2002) ≈ 15-35 ChrF (Popović, 2015) ≈ 40-70 BERTscore (Zhang et al, 2020) ≈ 60-80 COMET (Rei et al, 2020b) ≈ 15-60 BLEURT (Sellam et al, 2020) ≈ 25-50 METEOR (Denkowski and Lavie, 2011) ≈ 20-50 Fluency (Reeder, 2004) ≈ 1.0-3.0 The metrics and accompanying scores in Table 1 are meant to serve as a guide for what a company could expect from a LRMTS given the current systems that have been deployed in the wild. Most LRMTS are not good enough to use in the eyes of the low-resource community (Mager et al, 2023) but deployment can be considered for some cases like crises or others (O'Brien and Cadwell, 2017) as long as the proper care is taken to set appropriate expectations (especially for non-critical situations).…”

Section: Metricmentioning

confidence: 99%

See 1 more Smart Citation

A Research-Based Guide for the Creation and Deployment of a Low-Resource Machine Translation System

Ortega,

Church

2023

Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Proce

View full text Add to dashboard Cite

The machine translation (MT) field seems to focus heavily on English and other high-resource languages. Though, low-resource MT (LRMT) is receiving more attention than in the past. Successful LRMT systems (LRMTS) should make a compelling business case in terms of demand, cost and quality in order to be viable for end users. When used by communities where lowresource languages are spoken, LRMT quality should not only be determined by the use of traditional metrics like BLEU, but it should also take into account other factors in order to be inclusive and not risk overall rejection by the community. MT systems based on neural methods tend to perform better with high volumes of training data, but they may be unrealistic and even harmful for LRMT. It is obvious that for research purposes, the development and creation of LRMTS is necessary. However, in this article, we argue that two main workarounds could be considered by companies that are considering deployment of LRMTS in the wild: human-in-the-loop and sub-domains.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Metricmentioning

confidence: 99%

A Research-Based Guide for the Creation and Deployment of a Low-Resource Machine Translation System

Ortega,

Church

2023

Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Proce

View full text Add to dashboard Cite

show abstract

“…et al (2000) classified accuracy into several categories including simple string accuracy, generation string accuracy, and two corresponding tree-based accuracy. Reeder (2004) found the correlation between fluency and the number of words it takes to distinguish between human translation and MT output.…”

Section: Fluency Adequacy and Comprehensionmentioning

confidence: 99%

Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

Han,

Jones,

Smeaton

2021

Preprint

View full text Add to dashboard Cite

To facilitate effective translation modeling and translation studies, one of the crucial questions to address is how to assess translation quality. From the perspectives of accuracy, reliability, repeatability and cost, translation quality assessment (TQA) itself is a rich and challenging task. In this work, we present a high-level and concise survey of TQA methods, including both manual judgement criteria and automated evaluation metrics, which we classify into further detailed sub-categories. We hope that this work will be an asset for both translation model researchers and quality assessment researchers. In addition, we hope that it will enable practitioners to quickly develop a better understanding of the conventional TQA field, and to find corresponding closely relevant evaluation solutions for their own needs. This work may also serve inspire further development of quality assessment and evaluation methodologies for other natural language processing (NLP) tasks in addition to machine translation (MT), such as automatic text summarization (ATS), natural language understanding (NLU) and natural language generation (NLG). 1

show abstract

Investigation of Intelligibility Judgments

Cited by 2 publications

References 4 publications

A Research-Based Guide for the Creation and Deployment of a Low-Resource Machine Translation System

A Research-Based Guide for the Creation and Deployment of a Low-Resource Machine Translation System

Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

Contact Info

Product

Resources

About