N-gram posterior probability confidence measures for statistical machine translation: an empirical study

Abstract

We report an empirical study of n-gram posterior probability confidence measures for statistical machine translation (SMT). We first describe an efficient and practical algorithm for rapidly computing n-gram posterior probabilities from large translation word lattices. These probabilities are shown to be a good predictor of whether or not the n-gram is found in human reference translations, motivating their use as a confidence measure for SMT. Comprehensive n-gram precision and word coverage measurements are presented for a variety of different language pairs, domains and conditions. We analyze the effect on reference precision of using single or multiple references, and compare the precision of posteriors computed from k-best lists to those computed over the full evidence space of the lattice. We also demonstrate improved confidence by combining multiple lattices in a multi-source translation framework.

Confidence-based Rewriting of Machine Translation Output

“…Figure 2a shows that up to k = 10, 000 higher value of k to extract the rewriting phrase table increase the BLEU score on the test set. 12 We did not experiment with higher values of k, but plan to use the output lattice produced by 1-pass Moses to compute efficiently posteriors for larger sets of bi-phrases (de Gispert et al, 2013).…”

“…• phrase-based confidence score : bi-phrases are associated to a posterior probability, inspired from n-gram posterior probability estimation as defined in (de Gispert et al, 2013). Let E be the set of all hypotheses in the space of translation hypotheses defined by the n-best list used for source sentence f , and E α be the subset of E such that word alignments in sentence pairs (e , f ), ∀e ∈ E α , allow us to extract bi-phrase α.…”

“…Bach et al (2011) worked on the issue of predicting sentence-level and word-level MT errors by using WPP and other features derived from the source context, the source-target alignment, and dependency structures, but relied on a significantly large manually annotated corpus of MT errors. De Gispert et al (2013) calculate k-gram posterior probabilities from n-best lists or word lattices, and demonstrated that they were reasonably accurate indications of whether specific kgrams would be found or not in human reference translations. Finally, the work of Blackwood et al (2010) proposed to segment translation lattices according to confidence measures over the maximum likelihood translation hypothesis to focus on regions with potential translation errors.…”

LIMSI's Contribution to the WMT'16 Biomedical Translation Task

“…Each arc labelled u receives a score equal to the posterior unigram probability P (u|ε) of the system generating u at this position. P (u|ε) is computed as in (de Gispert et al, 2013):…”

Proceedings of the Ninth Workshop on Statistical Machine Translation

“…The lattices representing the search space considered to generate these pseudo-references also allow us to estimate the posterior probability of a target word that quantifies the probability that it is part of the system output (Gispert et al, 2013). Posteriors aggregate two pieces of information for each word in the final hypothesis: first, all the paths in the lattice (i.e.…”