BEER: BEtter Evaluation as Ranking

Stanojević, Miloš; Sima’an, Khalil

doi:10.3115/v1/w14-3354

Cited by 66 publications

(55 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because our method involves transliteration, which is applied at a character level, we found it also useful to evaluate the output with character-based metrics, which reward some translations even if the morphology is not completely correct. For this reason, we additionally report BEER (Stanojević and Sima'an 2014) and chrF3 (Popović 2015) scores.…”

Section: Neural Machine Translation Systemmentioning

confidence: 99%

Neural machine translation for low-resource languages without parallel corpora

Karakanta

Dehdari

Genabith

2017

Machine Translation

View full text Add to dashboard Cite

The problem of a total absence of parallel data is present for a large number of language pairs and can severely detriment the quality of machine translation. We describe a language-independent method to enable machine translation between a low-resource language (LRL) and a third language, e.g. English. We deal with cases of LRLs for which there is no readily available parallel data between the low-resource language and any other language, but there is ample training data between a closelyrelated high-resource language (HRL) and the third language. We take advantage of the similarities between the HRL and the LRL in order to transform the HRL data into data similar to the LRL using transliteration. The transliteration models are trained on transliteration pairs extracted from Wikipedia article titles. Then, we automatically back-translate monolingual LRL data with the models trained on the transliterated HRL data and use the resulting parallel corpus to train our final models. Our method achieves significant improvements in translation quality, close to the results that can be achieved by a general purpose neural machine translation system trained on a significant amount of parallel data. Moreover, the method does not rely on the existence of any parallel data for training, but attempts to bootstrap already existing resources in a related language.

show abstract

Section: Neural Machine Translation Systemmentioning

confidence: 99%

Neural machine translation for low-resource languages without parallel corpora

Karakanta

Dehdari

Genabith

2017

Machine Translation

View full text Add to dashboard Cite

show abstract

“…Following standard practice, we tune on BLEU, and after tuning we use the configuration with the highest scores on the development set with actual (corpus-level) BLEU evaluation. We report lowercase BLEU (Papineni et al 2002), METEOR (Denkowski and Lavie 2011), BEER (Stanojević and Sima'an 2014) and TER (Snover et al 2006) scores for the test set. We also report average translation length as a percentage of the reference length for all systems.…”

Section: Experimental Structurementioning

confidence: 99%

“…Nevertheless, both contribute evidence to the thesis that word order can be significantly improved without using syntax. Stanojević and Sima'an (2014) propose a new and highly successful machine translation evaluation method called BEER. This metric uses a multitude of weighted features, with weights that are directly trained to maximize correlation with human ranking.…”

Section: Learning Labelsmentioning

confidence: 99%

Labeling hierarchical phrase-based models without linguistic resources

Wenniger

Sima’an

2015

Machine Translation

View full text Add to dashboard Cite

Long-range word order differences are a well-known problem for machine translation. Unlike the standard phrase-based models which work with sequential and local phrase reordering, the hierarchical phrase-based model (Hiero) embeds the reordering of phrases within pairs of lexicalized context-free rules. This allows the model to handle long range reordering recursively. However, the Hiero grammar works with a single nonterminal label, which means that the rules are combined together into derivations independently and without reference to context outside the rules themselves. Follow-up work explored remedies involving nonterminal labels obtained from monolingual parsers and taggers. As of yet, no labeling mechanisms exist for the many languages for which there are no good quality parsers or taggers. In this paper we contribute a novel approach for acquiring reordering labels for Hiero grammars directly from the word-aligned parallel training corpus, without use of any taggers or parsers. The new labels represent types of alignment patterns in which a phrase pair is embedded within larger phrase pairs. In order to obtain alignment patterns that generalize well, we propose to decompose word alignments into trees over phrase pairs. Beside this labeling approach, we contribute coarse and sparse features for learning soft, weighted label-substitution as opposed to standard substitution. We report extensive experiments comparing our model to two baselines: Hiero and the known syntax augmented machine translation (SAMT) variant, which labels Hiero rules with nonterminals extracted from monolingual syntactic parses. We also test a simplified labeling scheme based on inversion transduction grammar (ITG English task we obtain performance improvement up to 1 BLEU point, whereas for the German-English task, where morphology is an issue, a minor (but statistically significant) improvement of 0.2 BLEU points is reported over SAMT. While ITG labeling does give a performance improvement, it remains sometimes suboptimal relative to our proposed labeling scheme.

show abstract

“…Many metrics have been proposed for MT that compare system translations against human references, with the most popular being BLEU (Papineni et al, 2002), METEOR (Denkowski and Lavie, 2014), TER (Snover et al, 2006), and, more recently, BEER (Stanojevic and Sima'an, 2014). These and other automatic metrics are often criticised for providing scores that can be non-intuitive and uninformative, especially at the sentence level (Zhang et al, 2004;Song et al, 2013;Babych, 2014).…”

Section: Introductionmentioning

confidence: 99%

Vis-Eval Metric Viewer: A Visualisation Tool for Inspecting and Evaluating Metric Scores of Machine Translation Output

Steele¹,

Specia²

2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: De

View full text Add to dashboard Cite

Machine Translation systems are usually evaluated and compared using automated evaluation metrics such as BLEU and METEOR to score the generated translations against human translations. However, the interaction with the output from the metrics is relatively limited and results are commonly a single score along with a few additional statistics. Whilst this may be enough for system comparison it does not provide much useful feedback or a means for inspecting translations and their respective scores. Vis-Eval Metric Viewer (VEMV) is a tool designed to provide visualisation of multiple evaluation scores so they can be easily interpreted by a user. VEMV takes in the source, reference, and hypothesis files as parameters, and scores the hypotheses using several popular evaluation metrics simultaneously. Scores are produced at both the sentence and dataset level and results are written locally to a series of HTML files that can be viewed on a web browser. The individual scored sentences can easily be inspected using powerful search and selection functions and results can be visualised with graphical representations of the scores and distributions.

show abstract

BEER: BEtter Evaluation as Ranking

Cited by 66 publications

References 11 publications

Neural machine translation for low-resource languages without parallel corpora

Neural machine translation for low-resource languages without parallel corpora

Labeling hierarchical phrase-based models without linguistic resources

Vis-Eval Metric Viewer: A Visualisation Tool for Inspecting and Evaluating Metric Scores of Machine Translation Output

Contact Info

Product

Resources

About