Improving Search using Proximity-Based Statistics

Abstract

Modern retrieval systems often use more sophisticated ranking models. Although more new features are added in, term proximity has been studied for a long time, and still plays an important role. A recent study by Huston and Croft [2] shows that many-term dependency is a better choice for a large corpus and long queries. However, utilizing proximity-based features often leads to computational overhead, and most of the existing solutions are tailored to term pairs. Fewer studies have focused on many-term proximity computation, and the plane-sweep approach proposed by Sadakane and Imai [6] is still state-of-the-art. Consider a multi-pass retrieval process where the proximity features could be an effective first pass ranker if we can reduce the cost of the proximity calculation.In this PhD project, we consider the following questions: (i) How important are the proximity statistics in the term dependency models and what is the cost of extracting the proximity features? (ii) Although all term dependencies are considered in ranking models, can we design an early termination strategy considering only partial proximity? Moreover, instead of viewing the term from the same level, can we utilizing its locality for obtaining more efficiency? (iii) How do we best organize the term proximity statistics to be more indexable, facilitating the extraction process? (iv) How do we best define the approximation form of term proximity in order to find the best trade-off between effectiveness and efficiency?In a preliminary experimental study, Lu et al. [3] compare how different term dependency components affect the entire ranking models show that although the phrase component helps to improve the effectiveness in an overall sense, it degrades dramatically on