Understanding and Predicting Graded Search Satisfaction

Jiang, Jiepu; Awadallah, Ahmed Hassan; Shi, Xiaolin; White, Ryen W.

doi:10.1145/2684822.2685319

Cited by 77 publications

(71 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since satisfaction is important for both search engine evaluation and optimization, a number of research studies have tried to quantify user satisfaction in both desktop search [20,47] and mobile search [26,28], and in both homogeneous [31] and heterogeneous search environment [10]. In recent years, a number of works (e.g, [21,22]) have started using the bene t-cost framework to analyze the satisfaction judgement process of users. In this framework, both the bene t factors (document relevance) and cost factors (the e ort users spend on examining search engine result pages (SERPs) and landing pages) are used to estimate satisfaction.…”

Section: Search Satisfactionmentioning

confidence: 99%

“…-TimeToFirstClick, TimeToLastClick -Time delta between the start of search session and the rst click and last click in the session, respectively. -DsatClickCount, DsatClickRatio -Previous studies divide clicks into satis ed clicks and dissatis ed clicks based on various dwell time thresholds [17,21]. We tested di erent thresholds and choose to de ne clicks with a dwell time <15s as dissatis ed clicks because it performs the best on our dataset.…”

Section: Comparison Across Online Metricsmentioning

confidence: 99%

“…We tested di erent thresholds and choose to de ne clicks with a dwell time <15s as dissatis ed clicks because it performs the best on our dataset. Previous work [21] also analyzed the threshold of 15s to di erentiate satis ed clicks. We calculate the number and ratio of dissatis ed clicks, respectively.…”

Section: Comparison Across Online Metricsmentioning

confidence: 99%

“…is is reasonable because the o ine metrics measure the quality of search result page based on relevance assessments and users usually tend to feel more satis ed if the search results are of high quality [34]. On the contrary, the interactions signals which online metrics adopted usually re ects search e ort and high search e ort can reduce user satisfaction [10,21]. MaxScroll also correlates with satisfaction negatively because a longer scroll distance may also indicate more search e ort.…”

Section: Comparison Across Online Metricsmentioning

confidence: 99%

“…In addition, it is o en cheap and fast to collect such data in modern search engines, making it particularly easy to scale up online evaluation. Typical online metrics include clickbased metrics such as CTR (click through rate), UCTR [11] (binary value representing click) and PLC [9] (number of clicks divided by the position of the lowest click) as well as dwell time-based metrics such as query dwell time, average of click dwell time [21] and so on. Although easily scalable and arguably more truthful indication of operational IR e ectiveness, online metrics can su er from various biases present in typical search logs.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Chen

Zhou

Liu

et al. 2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

As in most information retrieval (IR) studies, evaluation plays an essential part in Web search research. Both o ine and online evaluation metrics are adopted in measuring the performance of search engines. O ine metrics are usually based on relevance judgments of query-document pairs from assessors while online metrics exploit the user behavior data, such as clicks, collected from search engines to compare search algorithms. Although both types of IR evaluation metrics have achieved success, to what extent can they predict user satisfaction still remains under-investigated. To shed light on this research question, we meta-evaluate a series of existing online and o ine metrics to study how well they infer actual search user satisfaction in di erent search scenarios. We nd that both types of evaluation metrics signi cantly correlate with user satisfaction while they re ect satisfaction from di erent perspectives for di erent search tasks. O ine metrics be er align with user satisfaction in homogeneous search (i.e. ten blue links) whereas online metrics outperform when vertical results are federated. Finally, we also propose to incorporate mouse hover information into existing online evaluation metrics, and empirically show that they be er align with search user satisfaction than click-based online metrics.

show abstract

Section: Search Satisfactionmentioning

confidence: 99%

Section: Comparison Across Online Metricsmentioning

confidence: 99%

Section: Comparison Across Online Metricsmentioning

confidence: 99%

Section: Comparison Across Online Metricsmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Chen

Zhou

Liu

et al. 2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

Understanding an enriched multidimensional user relevance model by analyzing query logs

Zhang

Song

et al. 2017

Asso for Info Science & Tech

View full text Add to dashboard Cite

Modeling multidimensional relevance in information retrieval (IR) has attracted much attention in recent years. However, most existing studies are conducted through relatively small-scale user studies, which may not reflect a real-world and natural search scenario. In this article, we propose to study the multidimensional user relevance model (MURM) on large scale query logs, which record users' various search behaviors (e.g., query reformulations, clicks and dwelling time, etc.) in natural search settings. We advance an existing MURM model (including five dimensions: topicality, novelty, reliability, understandability, and scope) by providing two additional dimensions, that is, interest and habit. The two new dimensions represent personalized relevance judgment on retrieved documents. Further, for each dimension in the enriched MURM model, a set of computable features are formulated. By conducting extensive document ranking experiments on Bing's query logs and TREC session Track data, we systematically investigated the impact of each dimension on retrieval performance and gained a series of insightful findings which may bring benefits for the design of future IR systems. IntroductionThere have been numerous attempts (Tombros, Ruthven, & Jose, 2005;Xu & Yin, 2008;Xu & Chen, 2006;Zhang, Zhang, Lease, & Gwizdka, 2014) to understand users' search behaviors when retrieving information with search engines, foe example, relevance judgment, satisfaction or dissatisfaction with search results. Understanding how users conduct relevance judgment and what factors influence users' satisfaction with the search results would help researchers design more effective retrieval models and better evaluation methodologies, aiming to further improve users' search experience.Judging the relevance (or utility) of a retrieved document with respect to a user issued query (representing the user's current information need) is a central task for search engines. A large number of studies (Barry, 1998;Tombros et al., 2005;Xu & Yin, 2008;Xu & Chen, 2006;Zhang et al., 2014) have revealed that there exist a range of complex factors (e.g., topicality, novelty, reliability, understandability, and scope) affecting users' perception of relevance for the retrieved documents. However, the existing work is mainly based on small scale user studies, which may not reflect users' natural search scenarios, and the relevance judgments involved were made in a static way that cannot capture the dynamics of a user's information need, search interest, and habit.To address aforementioned limitations, in this article, we propose to study and understand the multidimensional relevance through analyzing real query logs that record real world user interactions (e.g., query reformulations, clicks, and dwelling time, etc.) with the search engine, as an important supplement to the numerous existing work based on user studies. Specifically, we analyze how different factors affect the users' perception of relevance on retrieved documents within the framework of the Multidi...

show abstract

Neural Correlates of Realisation of Satisfaction in a Successful Search Process

Paisalnan

Moshfeghi

Pollick

2021

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

In a search process, searchers review documents to gather information relevant to their information need (IN). During this process, searchers may experience the satisfaction of their IN, indicating they have gathered adequate relevant information to answer their need. This complex concept of satisfaction is the ultimate goal of search systems. Most studies in Information Retrieval (IR) have been attempted to understand how searchers' needs are satisfied based on behavioural observation. However, the psychophysiological manifestation during the moment of satisfaction still remains unclear. Here, we use functional Magnetic Resonance (fMRI) to investigate which brain regions are involved during the moment of satisfaction. Twenty-six participants participated in the experiment, designed to represent a search process while being scanned. Our result shows the human brain regions involved during the moment of satisfaction. These findings provide an important step in unravelling the concept of satisfaction in a search process.

show abstract

Understanding and Predicting Graded Search Satisfaction

Cited by 77 publications

References 39 publications

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

Understanding an enriched multidimensional user relevance model by analyzing query logs

Neural Correlates of Realisation of Satisfaction in a Successful Search Process

Contact Info

Product

Resources

About