Pipeline Creation Language for Machine Translation

Abstract

A pipeline is a commonly used architecture in machine translation (MT). Statistical MT can require, for example, many computational steps during training and decoding which are best viewed as a pipeline. These pipelines can be tedious to construct and are error prone since little or no type checking is performed, and no clear interface is defined for a pipeline's components. At times this can manifest itself as components requiring knowledge of how a preceding component's output is formed in order to consume it. Moreover, collaboration and sharing of pipelines or components becomes difficult since the components themselves may need to be changed in order to be used by others.In order to alleviate these problems a specialised language has been designed called Pipeline Creation Language (PCL). PCL allows users to construct pipelines that have components with well defined and compatibility checked interfaces. Components can be defined in packages, so individual components or packages of components, including entire pipelines, can be shared and composed with others. PCL supports operators which allow components to be executed sequentially, in parallel, or conditionally.