Exploiting Representations from Statistical Machine Translation for Cross-Language Information Retrieval

Abstract

This work explores how internal representations of modern statistical machine translation systems can be exploited for cross-language information retrieval. We tackle two core issues that are central to query translation: how to exploit context to generate more accurate translations and how to preserve ambiguity that may be present in the original query, thereby retaining a diverse set of translation alternatives. These two considerations are often in tension since ambiguity in natural language is typically resolved by exploiting context, but effective retrieval requires striking the right balance. We propose two novel query translation approaches: the grammar-based approach extracts translation probabilities from translation grammars, while the decoder-based approach takes advantage of n-best translation hypotheses. Both are context-sensitive, in contrast to a baseline context-insensitive approach that uses bilingual dictionaries for word-by-word translation. Experimental results show that by "opening up" modern statistical machine translation systems, we can access intermediate representations that yield high retrieval effectiveness. By combining evidence from multiple sources, we demonstrate significant improvements over competitive baselines on standard crosslanguage information retrieval test collections. In addition to effectiveness, the efficiency of our techniques are explored as well. ACM Reference Format:Ferhan Ture and Jimmy Lin. 2014. Exploiting representations from statistical machine translation for crosslanguage information retrieval.

Assessing Query Translation Quality Using Back Translation in Hindi-English CLIR

“…In 2012, Ture [30] used an internal representation of MT system for query translation and found significant improvement in retrieved results.…”

Learning to Translate: A Query-Specific Combination Approach for Cross-Lingual Information Retrieval

“…A similar approach by Herbert et al (2011) uses Wikipedia to provide translations of certain phrases and entities, and combining that with the Google Translate MT system yields statistically significant improvements in English-to-German retrieval. More recently, Ture et al (2012) presented a more sophisticated translation approach using the internal representation of an MT system, and reported statistically significant improvements when a preretrieval combination was performed.…”

“…However, as stated previously, numerous studies suggest that certain methods work better on certain queries, collections, languages. In fact, when weights are optimized separately on each collection, they differ substantially across different collections (Ture et al, 2012). For monolingual retrieval, there has been a series of learningto-rank (LTR) papers that determine weights for query concepts (Bendersky et al, 2011), such that retrieval effectiveness is maximized.…”

“…While our approach can applied to any set of translation methods, we focus on three methods that have complementary strengths and that have shown promise in CLIR: word-based probabilistic translation, one-best MT, and n-best probabilistic MT. We briefly present our implementation of each method; more details can be found in earlier work (Darwish and Oard, 2003;Ture et al, 2012).…”

“…While there are numerous combination-of-evidence techniques for both mono-lingual and cross-lingual IR, recent work suggests that there is no one-size-fits-all solution. In fact, the optimal combination weights (i.e., weights assigned to each piece of evidence in a linear combination) differ greatly across queries, tasks, languages, and other variants (Ture et al, 2012;Berger and Savoy, 2007).…”