XMEANT: Better semantic MT evaluation without reference translations

Abstract

We introduce XMEANT-a new cross-lingual version of the semantic frame based MT evaluation metric MEANT-which can correlate even more closely with human adequacy judgments than monolingual MEANT and eliminates the need for expensive human references. Previous work established that MEANT reflects translation adequacy with state-of-the-art accuracy, and optimizing MT systems against MEANT robustly improves translation quality. However, to go beyond tuning weights in the loglinear SMT model, a cross-lingual objective function that can deeply integrate semantic frame criteria into the MT training pipeline is needed. We show that cross-lingual XMEANT outperforms monolingual MEANT by (1) replacing the monolingual context vector model in MEANT with simple translation probabilities, and (2) incorporating bracketing ITG constraints.

Estimation of Cross-Lingual News Similarities Using Text-Mining Methods

“…With respect to the length of the text, a textual similarity task could be further categorized into two sub-tasks. Prevalent methods for cross-lingual document (i.e., long text) similarity could be categorized into four aspects (Rupnik et al 2016), Dictionary-based approaches (Kudo et al 2004), Probabilistic topic model based approaches (Taghva et al 2005), Matrix factorization based approaches (Lo et al 2014), and Monolingual approaches.…”

CNRC at SemEval-2016 Task 1: Experiments in Crosslingual Semantic Textual Similarity

“…In addition to the flat lexical semantic feature, we use XMEANT (Lo et al, 2014), the crosslingual semantic frame based machine translation evaluation metric, for generating shallow structural semantic features. We use MATE (Björkelund et al, 2009) for Spanish shallow semantic parsing and SENNA (Collobert et al, 2011) for English shallow semantic parsing.…”

Syntax and Semantics in Quality Estimation of Machine Translation

“…These problems are exacerbated in MEANT due to the automatic nature of the two steps. More recently, Lo et al (2014) extend MEANT to ranking translations without a reference by using phrase translation probabilities for aligning semantic role fillers of the source and its translation.…”