Evaluating combinations of ranked lists and visualizations of inter-document similarity

Abstract

We are interested in how ideas from document clustering can be used to improve the retrieval accuracy of ranked lists in interactive systems. In particular, we are interested in ways to evaluate the eectiveness of such systems to decide how they might best be constructed. In this study, we construct and evaluate systems that present the user with ranked lists and a visualization of inter-document similarities. We ®rst carry out a user study to evaluate the clustering/ranked list combination on instance-oriented retrieval, the task of the TREC-6 Interactive Track. We ®nd that although users generally prefer the combination, they are not able to use it to improve eectiveness. In the second half of this study, we develop and evaluate an approach that more directly combines the ranked list with information from inter-document similarities. Using the TREC collections and relevance judgments, we show that it is possible to realize substantial improvements in eectiveness by doing so, and that although users can use the combined information eectively, the system can provide hints that substantially improve on the user's solo eort. The resulting approach shares much in common with an interactive application of incremental relevance feedback. Throughout this study, we illustrate our work using two prototype systems constructed for these evaluations. The ®rst, AspInQuery, is a classic information retrieval system augmented with a specialized tool for recording information about instances of relevance. The other system, Lighthouse, is a Web-based application that combines a ranked list with a portrayal of inter-document similarity. Lighthouse can work with collections such as TREC, as well as the results of Web search engines. Ó

Clustering-Based Visual Interfaces for Presentation of Web Search Results: An Empirical Investigation

“…The scarcity (Roussinov and Chen, 2001;Zamir and Etzioni, 1999), and the practical difficulty (Allan et al, 2001) of empirical studies on the usefulness of visual presentation of information, especially in the web domain, reinforce the contribution of the paper. The rest of the paper is organized as follows.…”

“…There is very little structure in the list display of web search results, making it difficult to browse them (Allan et al, 2001;Amento, Terveen, and Hill, 2000a;Amento et al, 2000b;Bharat and Henzinger, 1998;Chen and Dumais, 2000;Mukherjea, Hirata, and Hara 1998;Pratt and Fagan, 2000;Pratt, Hearst, and Fagan 1999;Roussinov and Chen, 2001). According to the cluster hypothesis, mutually similar documents will tend to be relevant to similar information needs (Van Rijsbersgen, 1979).…”

“…For broadly formulated search queries, such a list may contain thousands of documents. Previous research has suggested that search engine users are not likely to go beyond the top 20 to 30 documents on these lists before getting bored or frustrated, and subsequently quitting the search (Allan et al, 2001;Amento, Terveen, and Hill, 2000a;Amento et al, 2000b;Bharat and Henzinger, 1998;Chen and Dumais, 2000;Mukherjea, Hirata, and Hara, 1998;Pratt and Fagan, 2000;Pratt, Hearst, and Fagan, 1999;Roussinov and Chen, 2001). Given that more and more people are using search engines for a variety of information needs, there is a need for research to enhance the ranked-list presentation of search results.…”

Interactive information retrieval

“…Many authors point to the importance of individual types of interactive support, either for different tasks, different stages within a search, or different search activities (e.g., Allan, Leuski, Swan, & Byrd, 2001;Kim & Allan, 2002;McDonald, Lai, & Tait, 2001;McDonald & Tait, 2003;White & Ruthven, 2006). Systems that offer the wrong support can work against the decisions a searcher must make.…”

Related Factors of Document Classification Performance in a Highly Inflectional Language

“…Many researchers have studied efficient text classification algorithms: vector space classification for a medical document filtering system with an IDF factor [16]; Okapi's tf score [1]; a neural network system with an ICF factor and χ-square (χ 2 ) algorithm [8]; a rule-based disjunctive production method based on term frequencies and boolean combinations of co-occurrence terms [2]; a context sensitive rule-based boolean method [5]; semantic classification based on conceptual models (e.g. Lingsoft's Nptool is applied by [3]) [4]; rule-based classification features based on important words, called field association words [7]; and comparisons of various classification algorithms [5], [10], [13], [18].…”