A Methodology for Evaluating Aggregated Search Results

Abstract

Abstract. Aggregated search is the task of incorporating results from different specialized search services, or verticals, into Web search results. While most prior work focuses on deciding which verticals to present, the task of deciding where in the Web results to embed the vertical results has received less attention. We propose a methodology for evaluating an aggregated set of results. Our method elicits a relatively small number of human judgements for a given query and then uses these to facilitate a metric-based evaluation of any possible presentation for the query. An extensive user study with 13 verticals confirms that, when users prefer one presentation of results over another, our metric agrees with the stated preference. By using Amazon's Mechanical Turk, we show that reliable assessments can be obtained quickly and inexpensively.

Task complexity, vertical display and user interaction in aggregated search

“…Existing methods for vertical selection and presentation use machine learning to combine different types of predictive evidence: query-string features [2,4,5,19,23], vertical query-log features [2,4,5,11,23], vertical content features [2,4,5,11], and implicit feedback features from previous presentations of the vertical [11,23]. Model tuning and evaluation is typically done with respect to editorial relevance judgements [2,3,4,5,19] or, in a production environment, with respect to user-generated clicks and skips [11,23]. In the first case, users do not actively participate in the evaluation.…”

“…Most published research in aggregated search has focused on automatic methods for predicting which verticals to present (vertical selection) [4,5,11,19] and where in the Web results to present them (vertical presentation) [2,3,23]. Evaluation of these systems has typically been conducted by using editorial vertical relevance judgements as the gold standard [2,3,4,5,19], or by using user-generated clicks on vertical results as a proxy for relevance [11,23].…”

“…Evaluation of these systems has typically been conducted by using editorial vertical relevance judgements as the gold standard [2,3,4,5,19], or by using user-generated clicks on vertical results as a proxy for relevance [11,23]. While these studies have greatly advanced the state of the art in aggregated search techniques, because users are far removed from the evaluation, they have contributed little insight about how users' higher-level objectives influence their engagement with vertical search results.…”

Automatic Summarization on Aggregated Search Results

“…Here I took help of some of my friends and given the job of a assigning the relevance score to the retrieved link as the response for their particular desired query. Thus they acted as evaluator and assigned the relevance score as given below in the Table No.1.However, there are other methodologies discussed for evaluating aggregated search results [8].…”

Evaluating aggregated search pages

“…Recent attempts to evaluate the utility of the whole aggregated search page [3,17] consider the three key components of aggregated search (VS, IS, RP) together. Our work takes a similar holistic approach and proposes a general evaluation framework for measuring aggregated search page quality.…”