Exploring Robustness Enhancements for Logic-Based Passage Filtering

Abstract

Abstract. The use of logic in question answering (QA) promises better accuracy of results, better utilization of the document collection, and a straightforward solution for integrating background knowledge. However, the brittleness of the logical approach still hinders its breakthrough into applications. Several proposals exist for making logic-based QA more robust against erroneous results of linguistic analysis and against gaps in the background knowledge: Extracting useful information from failed proofs, embedding the prover in a relaxation loop, and fusion of logicbased and shallow features using machine learning (ML). In the paper, we explore the effectiveness of these techniques for logic-based passage filtering in the LogAnswer question answering system. An evaluation on factual question of QA@CLEF07 reveals a precision of 54.8% and recall of 44.9% when relaxation results for two distinct provers are combined. 1

Combining Logic and Machine Learning for Answering Questions

“…The first run, loga081dede, used only the native prover of the MultiNet toolkit for logical processing. The second run, loga082dede, used the 'OPT' combination of the MultiNet prover and the E-KRHyper prover described in [4]. The motivation for using more than one prover is that following several relaxation paths by applying multiple provers might increase the chance of discovering a good provable query fragment.…”

“…Finally, the irScore feature provides the retrieval score determined by Lucene. The machine learning approach used for reranking the retrieved snippets based on the shallow features is the same as in [3,4]. It was implemented using the Weka toolbench [5].…”

“…6 Robustness is gained by using relaxation: if a proof is not found within a time limit, then query literals are skipped until a proof of the remaining query succeeds. The resulting skip count indicates (non-)entailment [6,4]. For efficiency reasons, relaxation is stopped before all literals are proved or skipped; a maximum of 3 relaxation cycles was chosen for the QA@CLEF runs.…”

“…The resulting MultiNet representations are segmented into passages and stored in a Lucene-based retrieval module. 4 The following kinds of information are indexed:…”

LogAnswer IN QUESTION ANSWERING FORUMS

“…If a parse of the query fails, then the system is still able to find answer sentences based on robust techniques like predictive annotation (Prager et al, 2000). The robustness enhancing techniques that were developed for LogAnswer are described in (Glöckner and Pelzer, 2008).…”