Phrase Tagset Mapping for French and English Treebanks and Its Application in Machine Translation Evaluation

Han, Aaron Li-Feng; Wong, Derek F.; Chao, Lidia S.; He, Liangye; Li, Shuo; Zhu, Liling

doi:10.1007/978-3-642-40722-2_13

Cited by 6 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Assuming that similar grammatical structures should occur in both source and translations, Avramidis et al (2011) perform evaluation on source (German) and target (English) sentences employing the features of sentence length ratio, unknown words, phrase numbers including noun phrase, verb phrase and prepositional phrase. Other similar work using phrase similarity includes (Li et al, 2012) that uses noun phrases and verb phrases from chunking, (Echizen-ya and Araki, 2010) that only uses the noun phrase chunking in automatic evaluation, and (Han et al, 2013c) that designs a universal phrase tagset for French to English MT evaluation.…”

Section: Syntactic Similaritymentioning

confidence: 99%

Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

Han,

Jones,

Smeaton

2021

Preprint

View full text Add to dashboard Cite

To facilitate effective translation modeling and translation studies, one of the crucial questions to address is how to assess translation quality. From the perspectives of accuracy, reliability, repeatability and cost, translation quality assessment (TQA) itself is a rich and challenging task. In this work, we present a high-level and concise survey of TQA methods, including both manual judgement criteria and automated evaluation metrics, which we classify into further detailed sub-categories. We hope that this work will be an asset for both translation model researchers and quality assessment researchers. In addition, we hope that it will enable practitioners to quickly develop a better understanding of the conventional TQA field, and to find corresponding closely relevant evaluation solutions for their own needs. This work may also serve inspire further development of quality assessment and evaluation methodologies for other natural language processing (NLP) tasks in addition to machine translation (MT), such as automatic text summarization (ATS), natural language understanding (NLU) and natural language generation (NLG). 1

show abstract

Section: Syntactic Similaritymentioning

confidence: 99%

Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

Han,

Jones,

Smeaton

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Assuming that the similar grammatical structures should occur on both source and translations, (Avramidis et al, 2011) perform the evaluation on source (German) and target (English) sentence employing the features of sentence length ratio, unknown words, phrase numbers including noun phrase, verb phrase and prepositional phrase. Other similar works using the phrase similarity include the (Li et al, 2012) that uses noun phrase and verb phrase from chunking and (Echizen-ya and Araki, 2010) that only uses the noun phrase chunking in automatic evaluation and (Han et al, 2013a) that designs a universal phrase tagset for French to English MT evaluation.…”

Section: Syntactic Similaritymentioning

confidence: 99%

Machine Translation Evaluation Resources and Methods: A Survey

Han¹

2016

Preprint

View full text Add to dashboard Cite

We introduce the Machine Translation (MT) evaluation survey that contains both manual and automatic evaluation methods. The traditional human evaluation criteria mainly include the intelligibility, fidelity, fluency, adequacy, comprehension, and informativeness. The advanced human assessments include task-oriented measures, post-editing, segment ranking, and extended criteriea, etc. We classify the automatic evaluation methods into two categories, including lexical similarity scenario and linguistic features application. The lexical similarity methods contain edit distance, precision, recall, F-measure, and word order. The linguistic features can be divided into syntactic features and semantic features respectively. The syntactic features include part of speech tag, phrase types and sentence structures, and the semantic features include named entity, synonyms, textual entailment, paraphrase, semantic roles, and language models. The deep learning models for evaluation are very newly proposed. Subsequently, we also introduce the evaluation methods for MT evaluation including different correlation scores, and the recent quality estimation (QE) tasks for MT. This paper differs from the existing works (Dorr et al., 2009; EuroMatrix, 2007) from several aspects, by introducing some recent development of MT evaluation measures, the different classifications from manual to automatic evaluation measures, the introduction of recent QE tasks of MT, and the concise construction of the content.We hope this work will be helpful for MT researchers to easily pick up some metrics that are best suitable for their specific MT model development, and help MT evaluation researchers to get a general clue of how MT evaluation research developed. Furthermore, hopefully, this work can also shine some light on other evaluation tasks, except for translation, of NLP fields. 1

show abstract

“…To address this problem, some researchers design the unsupervised MT evaluation approaches without using reference translations, which is also called quality estimation of MT. For example, Han et al design an unsupervised MT evaluation for French and English translation using their developed universal phrase tagset [ 16 ] and explore the performances of machine learning algorithms, for example, conditional random fields, support vector machine and naïve Bayes in the word-level quality estimation task of English to Spanish translation without using golden references [ 17 ]. Gamon et al [ 18 ] conduct a research about reference-free MT evaluation approaches also at sentence level, which utilizes the linear and nonlinear combinations of language model and SVM classifier to find the badly translated sentences.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Quality Estimation Model for English to German Translation and Its Application in Extensive Supervised Evaluation

Han

Wong

Chao

et al. 2014

The Scientific World Journal

Self Cite

View full text Add to dashboard Cite

With the rapid development of machine translation (MT), the MT evaluation becomes very important to timely tell us whether the MT system makes any progress. The conventional MT evaluation methods tend to calculate the similarity between hypothesis translations offered by automatic translation systems and reference translations offered by professional translators. There are several weaknesses in existing evaluation metrics. Firstly, the designed incomprehensive factors result in language-bias problem, which means they perform well on some special language pairs but weak on other language pairs. Secondly, they tend to use no linguistic features or too many linguistic features, of which no usage of linguistic feature draws a lot of criticism from the linguists and too many linguistic features make the model weak in repeatability. Thirdly, the employed reference translations are very expensive and sometimes not available in the practice. In this paper, the authors propose an unsupervised MT evaluation metric using universal part-of-speech tagset without relying on reference translations. The authors also explore the performances of the designed metric on traditional supervised evaluation tasks. Both the supervised and unsupervised experiments show that the designed methods yield higher correlation scores with human judgments.

show abstract

Phrase Tagset Mapping for French and English Treebanks and Its Application in Machine Translation Evaluation

Cited by 6 publications

References 4 publications

Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

Machine Translation Evaluation Resources and Methods: A Survey

Unsupervised Quality Estimation Model for English to German Translation and Its Application in Extensive Supervised Evaluation

Contact Info

Product

Resources

About