A tool for comparative evaluation in an interactive environment

Abstract

IR researchers need to collect comparative data to evaluate the worth of possible system improvements, for example new techniques for query enhancement. However it is sometimes difficult to achieve strict comparability in experiments designed for an interactive environment. We describe a method for merging results from different queries and collecting relevance feedback from users in comparative experiments. The method has been implemented in scripts supporting a Web-based front-end to a probabilistic retrieval system, using a relational database to log results. Its practical use is demonstrated in an investigation of initial query enhancement with collocate terms.