Reflections on Mira: Interactive evaluation in information retrieval

Abstract

Evaluation in information retrieval (IR) has focussed largely on noninteractive evaluation of text retrieval systems. This is increasingly at odds with how people use modern IR systems: in highly interactive settings to access linked, multimedia information. Furthermore, this approach ignores potential improvements through better interface design. In 1996, the Commission of the European Union Information Technologies Programme funded a 3‐year working group, Mira, to discuss and advance research in the area of evaluation frameworks for interactive and multimedia IR applications. Led by Keith van Rijsbergen, Steve Draper, and myself from Glasgow University, this working group brought together many of the leading researchers in the evaluation domain from both the IR and human–computer interaction (HCI) communities. This article presents my personal view of the main lines of discussion that took place throughout Mira: importing and adapting evaluation techniques from HCI, evaluating at different levels as appropriate, evaluating against different types of relevance and the new challenges that drive the need for rethinking the old evaluation approaches. The article concludes that we need to consider more varied forms of evaluation to complement engine evaluation.

An adaptive technique for content-based image retrieval

“…The user's judgement of relevance is naturally based on their current context, their preferences, and also their way of judging the semantic content of the images (e.g. [23,7]). Initially, the system uses the low-level image features as a quick way to 'estimate' the relevance values of the images.…”

Untitled

“…speed, layout, clearness, iteration (Robertson & HancockBeaulieu 1992, Saracevic 1995, Dunlop 2000. Therefore the data collected was rich for both 10 subjective and objective measures.…”

The CLEF Cross Language Image Retrieval Track (ImageCLEF) 2004

“…User-centred evaluation is important to assess the overall success of a retrieval system which takes into account other factors other than just system performance, e.g. the design of the user interface and system speed (Dunlop argues this in [7]). A num-ber of researchers have highlighted the advantages of user-centred evaluation, particularly in image retrieval systems (see, e.g.…”

“…A num-ber of researchers have highlighted the advantages of user-centred evaluation, particularly in image retrieval systems (see, e.g. [10], [14] and [7]). One of the main aims of ImageCLEF is to provide both the CLIR and image retrieval communities a number of useful resources (datasets and relevance assessments) to facilitate and promote further research in multilingual image retrieval.…”