An Initial Investigation into Fixed and Adaptive Stopping Strategies

Maxwell, David; Azzopardi, Leif; Järvelin, Kalervo; Keskustalo, Heikki

doi:10.1145/2766462.2767802

Cited by 15 publications

(31 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In previous work, we performed an initial analysis comparing a fixed depth stopping strategy against two implementations of the frustration point/disgust rules [23]. The fixed depth stopping strategy assumed that the searcher would stop at a particular document n in the ranked list, regardless of whether previous documents were considered relevant or non-relevant.…”

Section: Related Workmentioning

confidence: 99%

Searching and Stopping

Maxwell

Azzopardi

Järvelin

et al. 2015

Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

Searching naturally involves stopping points, both at a query level (how far down the ranked list should I go?) and at a session level (how many queries should I issue?). Understanding when searchers stop has been of much interest to the community because it is fundamental to how we evaluate search behaviour and performance. Research has shown that searchers find it di cult to formalise stopping criteria, and typically resort to their intuition of what is "good enough". While various heuristics and stopping criteria have been proposed, little work has investigated how well they perform, and whether searchers actually conform to any of these rules. In this paper, we undertake the first large scale study of stopping rules, investigating how they influence overall session performance, and which rules best match actual stopping behaviour. Our work is focused on stopping at the query level in the context of ad-hoc topic retrieval, where searchers undertake search tasks within a fixed time period. We show that stopping strategies based upon the disgust or frustration point rules -both of which capture a searcher's tolerance to non-relevance -typically result in (i) the best overall performance, and (ii) provide the closest approximation to actual searcher behaviour, although a fixed depth approach also performs remarkably well. Findings from this study have implications regarding how we build measures, and how we conduct simulations of search behaviours.

show abstract

Section: Related Workmentioning

confidence: 99%

Searching and Stopping

Maxwell

Azzopardi

Järvelin

et al. 2015

Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

show abstract

“…the slope of the user's gain curve. For example, a possible user computable measure of the current rate of gain could be the number of relevant documents found in the last 3 examined results or perhaps the number of relevant documents found in the last 2 minutes, or some of the strategies proposed by Maxwell et al [8]. White and Dumais [14] conducted a study and found that 24% of search engine switches were a result of dissatisfaction and 23% were because of an expectation of better results, and many of the other reasons reflect some desire for more or different information.…”

Section: Discussionmentioning

confidence: 99%

Modeling Optimal Switching Behavior

Smucker

Clarke

2016

Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval

View full text Add to dashboard Cite

Recently developed retrieval effectiveness measures have incorporated models of user behavior, but have limited themselves to predicting user performance over a single query and response. Accurate prediction of user performance with search systems must incorporate a means to model how users switch between different information sources. For example, a search session may consist of multiple queries with the user making decisions of when to switch from evaluating the current result list to a new result list produced by a query reformulation. Likewise, users may switch to a result list produced by a query suggestion or other interaction mechanism that produces a new search result list. In this paper, we simulate user behavior and investigate optimal switching behavior for a user who must decide when and if to issue their current query to another search engine. As a first step in understanding the problem space, we restrict our investigation and discussion to two top performing runs submitted to the TREC 2005 Robust track. We find four classes of switching behavior that a user would be faced with in making a decision about whether to switch from one result list to another.

show abstract

“…As done in prior simulations [2,19], the decision to click on a snippet -and the subsequent decision to mark a document relevant -are based upon interaction probabilities. The clicking (P (C)) and marking (P (M )) probabilities used here are reported in Table 1(b).…”

Section: Snippet and Document Decision Makingmentioning

confidence: 99%

Information Scent, Searching and Stopping

Maxwell

Azzopardi

2018

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. Current models and measures of the Interactive Information Retrieval (IIR) process typically assume that a searcher will always examine the first snippet in a given Search Engine Results Page (SERP), and then with some probability or cutoff, he or she will stop examining snippets and/or documents in the ranked list (snippet level stopping). Prior work has however shown that searchers will form an initial impression of the SERP, and will often abandon a page without clicking on or inspecting in detail any snippets or documents. That is, the information scent affects their decision to continue. In this work, we examine whether considering the information scent of a page leads to better predictions of stopping behaviour. In a simulated analysis, grounded with data from a prior user study, we show that introducing a SERP level stopping strategy can improve the performance attained by simulated users, resulting in an increase in gain across most snippet level stopping strategies. When compared to actual search and stopping behaviour, incorporating SERP level stopping offers a closer approximation than without. These findings show that models and measures that naïvely assume snippets and documents in a ranked list are actually examined in detail are less accurate, and that modelling SERP level stopping is required to create more realistic models of the search process.

show abstract

An Initial Investigation into Fixed and Adaptive Stopping Strategies

Cited by 15 publications

References 23 publications

Searching and Stopping

Searching and Stopping

Modeling Optimal Switching Behavior

Information Scent, Searching and Stopping

Contact Info

Product

Resources

About