The Design of Eman, an Experiment Manager

Bojar, Ondřej; Tamchyna, Aleš

doi:10.2478/pralin-2013-0003

Cited by 4 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our experiments are carried out on the CS-EN, DE-EN, and FR-EN language pairs using Eman, an experiment manager by Bojar and Tamchyna [88]. We evaluate our systems using BLEU [5] and PER (position-independent word error rate) [6].…”

Section: Machine Translation Experimentsmentioning

confidence: 99%

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pecina¹,

Dušek²,

Goeuriot³

et al. 2014

Artificial Intelligence in Medicine

Self Cite

View full text Add to dashboard Cite

Objective. We investigate machine translation (MT) of user search queries in the context of cross-lingual information retrieval (IR) in the medical domain. The main focus is on techniques to adapt MT to increase translation quality; however, we also explore MT adaptation to improve effectiveness of cross-lingual IR.Methods and Data. Our MT system is Moses, a state-of-the-art phrase-based statistical machine translation system. The IR system is based on the BM25 retrieval model implemented in the Lucene search engine. The MT techniques employed in this work include in-domain training and tuning, intelligent training data selection, optimization of phrase table configuration, compound splitting, and exploiting synonyms as translation variants. The IR methods include morphological normalization and using multiple translation variants for query expansion. The experiments are performed and thoroughly evaluated on three language pairs: Czech-English, German-English, and French-English. MT quality is evaluated on data sets created within the Khresmoi project and IR effectiveness is tested on the CLEF eHealth 2013 data sets.Results. The search query translation results achieved in our experiments are outstanding -our systems outperform not only our strong baselines, but also Google Translate and Microsoft Bing Translator in direct comparison carried out on all the language pairs. The baseline BLEU scores increased from 26.59 to 41.45 for Czech-English, from 23.03 to 40.82 for German-English, and from 32.67 to 40.82 for French-English. This is a 55% improvement on average. In terms of the IR performance on this particular test collection, a significant improvement over the baseline is achieved only for French-English. For Czech-English and German-English, the increased MT quality does not lead to better IR results.Conclusions. Most of the MT techniques employed in our experiments improve MT of medical search queries. Especially the intelligent training data selection proves to be very successful for domain adaptation of MT. Certain improvements are also obtained from German compound splitting on the source language side. Translation quality, however, does not appear to correlate with the IR performance -better translation does not necessarily yield better retrieval. We discuss in detail the contribution of the individual techniques and state-of-the-art features and provide future research directions.

show abstract

Section: Machine Translation Experimentsmentioning

confidence: 99%

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pecina¹,

Dušek²,

Goeuriot³

et al. 2014

Artificial Intelligence in Medicine

Self Cite

View full text Add to dashboard Cite

show abstract

“…An experiment manager ensures that each run of an experiment has a distinct location for its results. Experiment managers, such as Moses' EMS (Philipp Koehn, 2010) and Eman (Ondřej Bojar and Aleš Tamchyna, 2013), handle two concerns; the definition of a software pipeline, and collating results.…”

Section: Pcl and Experiments Managementmentioning

confidence: 99%

Pipeline Creation Language for Machine Translation

Johnson¹

2014

The Prague Bulletin of Mathematical Linguistics

View full text Add to dashboard Cite

A pipeline is a commonly used architecture in machine translation (MT). Statistical MT can require, for example, many computational steps during training and decoding which are best viewed as a pipeline. These pipelines can be tedious to construct and are error prone since little or no type checking is performed, and no clear interface is defined for a pipeline's components. At times this can manifest itself as components requiring knowledge of how a preceding component's output is formed in order to consume it. Moreover, collaboration and sharing of pipelines or components becomes difficult since the components themselves may need to be changed in order to be used by others.In order to alleviate these problems a specialised language has been designed called Pipeline Creation Language (PCL). PCL allows users to construct pipelines that have components with well defined and compatibility checked interfaces. Components can be defined in packages, so individual components or packages of components, including entire pipelines, can be shared and composed with others. PCL supports operators which allow components to be executed sequentially, in parallel, or conditionally.

show abstract

“…Eman (Bojar and Tamchyna, 2013) also has its roots in SMT research but is designed as a general framework for running scientific experiments. Its primary objectives are to avoid unnecessary recreation of intermediate results, and to ensure that all experiments are replicable by preserving and thoroughly documenting all experimental parameters and intermediate results.…”

Section: Introductionmentioning

confidence: 99%

“…The second approach is to write a dedicated build system, such as the Experiment Management System (EMS) for Moses (Koehn, 2010), or Experiment Manager (Eman), a more general framework for designing, running, and documenting scientific experiments (Bojar and Tamchyna, 2013).…”

Section: Introductionmentioning

confidence: 99%

Makefiles for Moses

Germann¹

2013

The Prague Bulletin of Mathematical Linguistics

View full text Add to dashboard Cite

Building MT systems with the Moses toolkit is a task so complex that it is rarely done manually. Over the years, several frameworks for building, running, and evaluating Moses systems have been developed, most notably the Experiment Management System (EMS). While EMS works well for standard experimental set-ups and offers good web integration, designing new experimental set-ups within EMS is not trivial, especially when the new processing pipeline differs considerably from the kind EMS is intended for. In this paper, I present M4M (Makefiles for Moses), a framework for building and evaluating Moses MT systems with the GNU Make utility. I illustrate the capabilities by a simple set-up that builds and compares two different systems with common resources. This set-up requires little more than putting training, tuning and evaluation data into the right directories and running Make.1 The purpose of this paper is twofold: to guide first-time users of Moses through the process of building baseline MT systems, and to discuss some lesser-known features of the Make utility that enable the MT practitioner to set up complex experimental scenarios efficiently. M4M is part of the Moses distribution.

show abstract

The Design of Eman, an Experiment Manager

Cited by 4 publications

References 11 publications

Adaptation of machine translation for multilingual information retrieval in the medical domain

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pipeline Creation Language for Machine Translation

Makefiles for Moses

Contact Info

Product

Resources

About