PARADIGM: Paraphrase Diagnostics through Grammar Matching

Abstract

Paraphrase evaluation is typically done either manually or through indirect, taskbased evaluation. We introduce an intrinsic evaluation PARADIGM which measures the goodness of paraphrase collections that are represented using synchronous grammars. We formulate two measures that evaluate these paraphrase grammars using gold standard sentential paraphrases drawn from a monolingual parallel corpus. The first measure calculates how often a paraphrase grammar is able to synchronously parse the sentence pairs in the corpus. The second measure enumerates paraphrase rules from the monolingual parallel corpus and calculates the overlap between this reference paraphrase collection and the paraphrase resource being evaluated. We demonstrate the use of these evaluation metrics on paraphrase collections derived from three different data types: multiple translations of classic French novels, comparable sentence pairs drawn from different newspapers, and bilingual parallel corpora. We show that PARADIGM correlates with human judgments more strongly than BLEU on a task-based evaluation of paraphrase quality.

Monolingual Phrase Alignment on Parse Forests

“…However, previous studies declared that the compositionality is violated in a monolingual phrase alignment (Burkett et al, 2010;Weese et al, 2014). Heilman and Smith (2010) discuss complex phrase reordering is prevalent in paraphrases and entailed text.…”

“…We used reference translations to evaluate machine translations 5 as sentential paraphrases (Weese et al, 2014). The reference translations of 10 to 30 words were extracted and paired, giving 41K pairs as a training corpus.…”

“…However, alignment quality was beyond their scope. Weese et al (2014) extracted SCFG from paraphrase corpora. They showed that parsing was only successful in 9.1% of paraphrases, confirming that a significant amount of transformations in paraphrases do not conform to compositionality or ITG.…”

The Geometry of Statistical Machine Translation

“…Word alignments and phrase-based conditional rule probabilities have been computed using MapReduce by Dyer et al [55]. Software for computing hierarchical phrase-based models using MapReduce is also readily available [97,147,152].…”

“…For each key in the file, its membership is tested in the query set. This is the approach adopted in the Joshua 3.0 decoder [152], which uses regular expressions or N-grams to test membership of translation rules. Venugopal and Zollmann [147] use MapReduce to scan a file of translation rules concurrently: a mapper is defined that tests if the vocabulary of a rule matches the vocabulary of a test set.…”