Findings of the 2014 Workshop on Statistical Machine Translation

Bojar, Ondřej; Buck, Christian; Federmann, Christian; Haddow, Barry; Koehn, Philipp; Leveling, Johannes; Monz, Christof; Pecina, Pavel; Post, Matt; Saint-Amand, Hervé; Soricut, Radu; Specia, Lucia; Tamchyna, Aleš

doi:10.3115/v1/w14-3302

Cited by 555 publications

(581 citation statements)

References 61 publications

Supporting

Mentioning

577

Contrasting

Unclassified

Order By: Relevance

“…Such scores are comparable to the state-of-the-art results reported recently on the same data sets [91]. The scores measured on the med domain are consistently higher by about 2 BLEU points (27.06-31.45).…”

Section: Baseline Translation Systemssupporting

confidence: 88%

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pecina¹,

Dušek²,

Goeuriot³

et al. 2014

Artificial Intelligence in Medicine

Self Cite

View full text Add to dashboard Cite

Objective. We investigate machine translation (MT) of user search queries in the context of cross-lingual information retrieval (IR) in the medical domain. The main focus is on techniques to adapt MT to increase translation quality; however, we also explore MT adaptation to improve effectiveness of cross-lingual IR.Methods and Data. Our MT system is Moses, a state-of-the-art phrase-based statistical machine translation system. The IR system is based on the BM25 retrieval model implemented in the Lucene search engine. The MT techniques employed in this work include in-domain training and tuning, intelligent training data selection, optimization of phrase table configuration, compound splitting, and exploiting synonyms as translation variants. The IR methods include morphological normalization and using multiple translation variants for query expansion. The experiments are performed and thoroughly evaluated on three language pairs: Czech-English, German-English, and French-English. MT quality is evaluated on data sets created within the Khresmoi project and IR effectiveness is tested on the CLEF eHealth 2013 data sets.Results. The search query translation results achieved in our experiments are outstanding -our systems outperform not only our strong baselines, but also Google Translate and Microsoft Bing Translator in direct comparison carried out on all the language pairs. The baseline BLEU scores increased from 26.59 to 41.45 for Czech-English, from 23.03 to 40.82 for German-English, and from 32.67 to 40.82 for French-English. This is a 55% improvement on average. In terms of the IR performance on this particular test collection, a significant improvement over the baseline is achieved only for French-English. For Czech-English and German-English, the increased MT quality does not lead to better IR results.Conclusions. Most of the MT techniques employed in our experiments improve MT of medical search queries. Especially the intelligent training data selection proves to be very successful for domain adaptation of MT. Certain improvements are also obtained from German compound splitting on the source language side. Translation quality, however, does not appear to correlate with the IR performance -better translation does not necessarily yield better retrieval. We discuss in detail the contribution of the individual techniques and state-of-the-art features and provide future research directions.

show abstract

Section: Baseline Translation Systemssupporting

confidence: 88%

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pecina¹,

Dušek²,

Goeuriot³

et al. 2014

Artificial Intelligence in Medicine

Self Cite

View full text Add to dashboard Cite

show abstract

“…Formiga et al (2013) confirm that ordinal regression makes better predictions as compared to ordering MT outputs, based on separate regression models over absolute scores of adequacy. When it comes to learning from ordinal rankings, Avramidis and Popović (2013) set the state-of-the-art performance for German-English, in the frame of a WMT shared task in QE (Bojar et al, 2013), Previous work has motivated the use of grammatical features focusing in specific structures (eg. Mutton et al, 2007), feature selection was motivated by Specia et al (2009), whereas an analysis of features was done by Felice and Specia (2012); nevertheless all the above work is limited to non-Comparative QE.…”

Section: Related Workmentioning

confidence: 99%

Comparative Quality Estimation for Machine Translation Observations on Machine Learning and Features

Avramidis

2017

The Prague Bulletin of Mathematical Linguistics

View full text Add to dashboard Cite

A deeper analysis on Comparative Quality Estimation is presented by extending the stateof-the-art methods with adequacy and grammatical features from other Quality Estimation tasks. The previously used linear method, unable to cope with the augmented features, is replaced with a boosting classifier assisted by feature selection. The methods indicated show improved performance for 6 language pairs, when applied on the output from MT systems developed over 7 years. The improved models compete better with reference-aware metrics.Notable conclusions are reached through the examination of the contribution of the features in the models, whereas it is possible to identify common MT errors that are captured by the features. Many grammatical/fluency features have a good contribution, few adequacy features have some contribution, whereas source complexity features are of no use. The importance of many fluency and adequacy features is language-specific.

show abstract

“…We tested HMEANT on a set of four best MT systems (Bojar et al, 2013) for the English-Russian language pair (Table 2).…”

Section: Test Setmentioning

confidence: 99%

Applying HMEANT to English-Russian Translations

Chuchunkov¹,

Tarelkin²,

Galinskaya³

2014

Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation

View full text Add to dashboard Cite

In this paper we report the results of first experiments with HMEANT (a semiautomatic evaluation metric that assesses translation utility by matching semantic role fillers) on the Russian language. We developed a web-based annotation interface and with its help evaluated practicability of this metric in the MT research and development process. We studied reliability, language independence, labor cost and discriminatory power of HMEANT by evaluating English-Russian translation of several MT systems. Role labeling and alignment were done by two groups of annotators -with linguistic background and without it. Experimental results were not univocal and changed from very high inter-annotator agreement in role labeling to much lower values at role alignment stage, good correlation of HMEANT with human ranking at the system level significantly decreased at the sentence level. Analysis of experimental results and annotators' feedback suggests that HMEANT annotation guidelines need some adaptation for Russian.

show abstract

Findings of the 2014 Workshop on Statistical Machine Translation

Cited by 555 publications

References 61 publications

Adaptation of machine translation for multilingual information retrieval in the medical domain

Adaptation of machine translation for multilingual information retrieval in the medical domain

Comparative Quality Estimation for Machine Translation Observations on Machine Learning and Features

Applying HMEANT to English-Russian Translations

Contact Info

Product

Resources

About