Test Collection-Based IR Evaluation Needs Extension toward Sessions – A Case of Extremely Short Queries

Abstract

Abstract. There is overwhelming evidence suggesting that the real users of IR systems often prefer using extremely short queries (one or two individual words) but they try out several queries if needed. Such behavior is fundamentally different from the process modeled in the traditional test collection-based IR evaluation based on using more verbose queries and only one query per topic. In the present paper, we propose an extension to the test collection-based evaluation. We will utilize sequences of short queries based on empirically grounded but idealized session strategies. We employ TREC data and have test persons to suggest search words, while simulating sessions based on the idealized strategies for repeatability and control. The experimental results show that, surprisingly, web-like very short queries (including one-word query sequences) typically lead to good enough results even in a TREC type test collection. This finding motivates the observed real user behavior: as few very simple attempts normally lead to good enough results, there is no need to pay more effort. We conclude by discussing the consequences of our finding for IR evaluation.

The economics in interactive information retrieval

“…While, for generated queries of length 3, the performance, in terms of MAP was 0.30, 0.1950 and 0.266, respectively. Also, on the whole the queries generated appeared quite sensible: and reflected the querying behavior observed in the study by Keskustalo et al [16], referred to as S3. S3 was the most common strategy they observed, and was where users would issue multiple queries of length three: pivoting on two key words, and then varying the third query term.…”

“…For example, in practice users often issues short queries [22], but longer queries have been shown to be more effective [14]. Kestalous et al [16] tried to justify this strategy empirically, and showed that a series of extremely short queries can be quite effective for finding one highly relevant document. But searching for a number of relevant documents often requires numerous queries to be posed during the search session [15].…”

Information Scent, Searching and Stopping

“…Query Generation Keskustalo et al [13] proposed a series of idealised, prototypical approaches to generating queries, as identified from a user study. In particular, strategy QS3 identified by the authors offered reasonably good performance.…”

“…7 Two pivot terms were selected, with an additional third term added. However, users have been shown to steadily build up their queries as they acquire more information, first issuing short queries then increasing their length [13]. To this end, we created a modified querying strategy, taking the pivot terms, issuing these as individual queries first, and then combining them as the pivot.…”

“…To this end, we created a modified querying strategy, taking the pivot terms, issuing these as individual queries first, and then combining them as the pivot. This approach: (i) makes the querying strategy more realistic [13]; and (ii) allows us to test the robustness of both the SERP level and snippet level stopping strategies when faced with both good and poor performing queries. 8 …”