Aspect windows, 3-D visualizations, and indirect comparisons of information retrieval systems

Abstract

We built two Information Retrieval systems that were targeted for the TREC-6 "aspect oriented" retrieval track. The systems were built to test the usefulness of different visualizations in an interactive IR setting-in particular, an "aspect window" for the chosen task, and a 3-D visualization of document inter-relationships. We studied 24 users of the system in order to investigate: whether the systems were more effective than a control system, whether experienced users outperformed novices, whether spatial reasoning ability was a good predictor of effective use of 3-D, and whether the systems could be compared indirectly via a control system. Our results show substantial differences in user performance are related to spatial reasoning ability and to a lesser degree other traits. We also obtained markedly different results from the direct and indirect comparisons. IntroductionWe are interested in building and evaluating high quality information retrieval and organization tools. We believe that effective use of such tools may require talented users or significant amounts of training. There are many settings where experts in the field are required to spend time learning a tool-eg., CAD/CAM applications, statistical analysis packages-and the gains from learning the system more'than outweigh the time spent learning it. Novice users may iind such systems puzzling, but we do not feel that diminishes the value of a targeted system. Further, other researchers are investigating the usefulness of systems for users with little to no searching experience[23, 31.On the other hand, we have no interest in building systems that are inherently difficult to use. Indeed, the better and easier to use a system's underlying design is, the more complexity we can introduce without overburdening the user [21]. For that reason, we are interested in basic issues in interactive computing, among them: how effective are simple system features, how can we compare our various systems, and are there any measures we can Interactive Track, an evaluation of 'aspect oriented information retrieval," wherein users are tasked with identifying as many %s-pects" of relevance to a query as they can. For example, in a query about ferry sinkings in the news, the task was to find a list of all ferries that sank, not to find all documents about ferry sinkings. The structure of our experiments was determined to a large extent by the TREC-6 guidelines; they are explained in more detail below. Because of our interests in targeted systems, we chose to build and evaluate a system that was designed specifically to aid a user with aspect retrieval. The alternative would have been to use a vanilla search engineperhaps slightly enhanced to look at some specific search technique-for the task; we felt that approach would not sufficiently address our interests. At the same time, we have been investigating 3-D visualizations of document relatedness (clustering), so we chose to create a slightly enhanced version of our system that included a 3-D visualization.The questions...

The effect of cognitive abilities on information search for tasks of varying levels of complexity

“…There results indicated that spatial orientation was significantly related to search performance in the navigationally-intense condition (i.e., map condition). Swan and Allan [47] found that visualization ability was not related to users' success using a 3-D interface.…”

Grouper: a dynamic clustering interface to Web search results

“…This can help the user get an overview of the collection, or, alternatively, find interesting documents faster, as once one document has been found it is easy to find others similar to it in the visualized space. There are four major techniques to visualize inter-document similarities: document networks [8,31], spring embeddings [3,30], document clustering [1,13,18,38] and self-organizing maps [16,20,22]. Of the four major techniques, only document clustering appears to be both fast enough and intuitive enough to require little training or adjustment time from the user [13,37].…”

Toward a semantic granularity model for domain-specific information retrieval

“…"Aspect Retrieval" was explored in the interactive track of TREC-6 [Lagergren and Over 1998;Over 1997;Swan and Allan 1998], TREC-7 [Belkin et al 1998;Bodner and Chignell 1998;Fuller et al 1998;Gey et al 1998;Hersh et al 1998;Ogden et al 1998;Over 1998;Robertson et al 1998;Yang et al 1998], and TREC-8 [Beaulieu et al 1999;Belkin et al 1999;Fuller et al 1999;Hersh 1999;Hersh et al 1999;Larson 1999;Yang et al 1999]. Aspect retrieval helps information seekers retrieve documents covering as many different aspects of an information topic as possible given a limited time span.…”

“…Aspect retrieval helps information seekers retrieve documents covering as many different aspects of an information topic as possible given a limited time span. An aspect is defined as one of the many possible answers to an information topic [Over 1998;Swan and Allan 1998]. For every document in a collection, the corresponding aspects of topics are judged by human assessors to create the correct answers and to evaluate the performance of various aspect retrieval methods.…”